Metraipolis › archive › dc_8d20efacb16e
Agreement rather than normalisation keeps depth reusable in looped language models — arXiv:2610.31943
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.31943
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-26T19:53:46.839Z
- SHA-256 of the archived text
d22ce32fef7b531b127a7a47153efdcfe15adaa44103c702a54f0d7ff0a64086- Stable URL of this page
https://metraipolis.com/documento/dc_8d20efacb16e
How to cite
Agreement rather than normalisation keeps depth reusable in looped language models — arXiv:2610.31943 — arXiv (en), https://arxiv.org/abs/2610.31943 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d20efacb16e (Metraipolis archive dc_8d20efacb16e, sha256 d22ce32fef7b531b127a7a47153efdcfe15adaa44103c702a54f0d7ff0a64086)
BibTeX
@misc{dc8d20efacb16e,
title = {Agreement rather than normalisation keeps depth reusable in looped language models — arXiv:2610.31943},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.31943},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d20efacb16e; sha256 d22ce32fef7b531b127a7a47153efdcfe15adaa44103c702a54f0d7ff0a64086},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31943. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Looped language models apply one block of layers repeatedly, and the design pays off only if depth is reusable: loop steps beyond the trained budget must remain usable at inference. We ask whether that rests on normalising the state between loop steps, as the released Ouro models do, or on agreement between the loop a model is trained with and the loop it is run with. On released weights this loop-boundary normalisation can be removed without adding parameters and with the readout path bit-identical, and the operation is an identity at one loop step, which serves as an implementation check. Removing it takes Ouro-1.4B and Ouro-2.6B to zero accuracy by the trained budget on a balanced binary task (chance 0.500). State-norm growth, the failure that prior accounts predict, is not the cause: the norm stays within a factor of 2.021 while the head commits to a single wrong token. Yet in 19M-parameter models trained from scratch on two synthetic tasks, 16 seeds per configuration, training without the normalisation is never worse than training with it and extrapolates better: the two are closest at the trained depth and separate on both sides of it (Cliff's delta up to 0.867 beyond the budget). Both results follow if agreement, and not the normalisation itself, is what keeps depth reusable. The damage sits in the readout together with the last layer of the block: in the models trained from scratch, refitting the head over the frozen modified loop recovers a third of the lost accuracy and plateaus, while refitting the last block layer with it recovers all of it. Finally, comparing looped with stacked depth at an equal inference loop count is confounded by unequal overshoot of the trained budget; at matched overshoot the advantage of looping vanishes on one task and shrinks on the other. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.31943 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-26T19:45:48.383Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).