Metraipolis › archive › dc_12036ec01622

Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining — arXiv:2611.06206

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.06206
License (as archived)
CC0-1.0
Archived
2026-11-05T18:18:44.664Z
SHA-256 of the archived text
ccc2370faeed23894c0027cacd21ce2a33539c47d652890a090d062ed4dba7d3
Stable URL of this page
https://metraipolis.com/documento/dc_12036ec01622

How to cite

Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining — arXiv:2611.06206 — arXiv (en), https://arxiv.org/abs/2611.06206 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_12036ec01622 (Metraipolis archive dc_12036ec01622, sha256 ccc2370faeed23894c0027cacd21ce2a33539c47d652890a090d062ed4dba7d3)
BibTeX
@misc{dc12036ec01622,
  title = {Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining — arXiv:2611.06206},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06206},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_12036ec01622; sha256 ccc2370faeed23894c0027cacd21ce2a33539c47d652890a090d062ed4dba7d3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06206. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multimodal models have often scored below text-only baselines in the BabyLM Challenge (Klerings et al., 2024; Takmaz et al., 2025). We test whether additional recurrent computation helps close this gap. Recurrence increases computation by reusing layers without adding parameters or training data. Using a single implementation, we compare standard and looped Transformers under a 100M-word budget, with three seeds per main condition. Each image is represented by one feature vector from a frozen DINOv2 encoder, used as a prefix token. Caption words count toward the budget, so multimodal training also changes the language data. In a pre-registered text-only comparison, a 12-layer stack repeated four times exceeds a standard 48-layer Transformer by 5.4 BLiMP percentage points at matched inference compute. However, multimodal training lowers BLiMP accuracy by about 3.8 points in both architectures, with no clear mitigation from recurrence. An exploratory loop-side no-image control, trained with the multimodal language data and batching schedule, scores 5.7 points below the text-only baseline. Adding the visual prefix raises accuracy by 1.85 points on average, but this contrast is uncertain with three seeds. These results suggest that changes in language data and its presentation are a major source of the drop. Diagnostic tests show that the models use visual information; its contribution to grammatical generalization remains uncertain under this single-vector interface.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.06206 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:13:03.943Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).