Metraipolis › archive › dc_6a33a69ada8d
What Makes Recurrence Effective in Looped Language Models? — arXiv:2609.36636
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36636
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:57:28.563Z
- SHA-256 of the archived text
b219614789a72cc72cd682ce2b632a74955461f5fb4098af625cdcc3896bb0c6- Stable URL of this page
https://metraipolis.com/documento/dc_6a33a69ada8d
How to cite
What Makes Recurrence Effective in Looped Language Models? — arXiv:2609.36636 — arXiv (en), https://arxiv.org/abs/2609.36636 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6a33a69ada8d (Metraipolis archive dc_6a33a69ada8d, sha256 b219614789a72cc72cd682ce2b632a74955461f5fb4098af625cdcc3896bb0c6)
BibTeX
@misc{dc6a33a69ada8d,
title = {What Makes Recurrence Effective in Looped Language Models? — arXiv:2609.36636},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36636},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6a33a69ada8d; sha256 b219614789a72cc72cd682ce2b632a74955461f5fb4098af625cdcc3896bb0c6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36636. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.36636 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:27:03.407Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).