Metraipolis › archive › dc_db593020a58f

Closing the Loop: Practical Training Recipes for Looped Language Models — arXiv:2610.00673

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00673
License (as archived)
CC0-1.0
Archived
2026-10-06T12:10:41.412Z
SHA-256 of the archived text
4da60249621a399f52907f92107520e6a12c92131437cea3d633b8f2e67db748
Stable URL of this page
https://metraipolis.com/documento/dc_db593020a58f

How to cite

Closing the Loop: Practical Training Recipes for Looped Language Models — arXiv:2610.00673 — arXiv (en), https://arxiv.org/abs/2610.00673 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_db593020a58f (Metraipolis archive dc_db593020a58f, sha256 4da60249621a399f52907f92107520e6a12c92131437cea3d633b8f2e67db748)
BibTeX
@misc{dcdb593020a58f,
  title = {Closing the Loop: Practical Training Recipes for Looped Language Models — arXiv:2610.00673},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00673},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_db593020a58f; sha256 4da60249621a399f52907f92107520e6a12c92131437cea3d633b8f2e67db748},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00673. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain difficult to separate from differences in data and training. In this work, we establish practical training recipes for looped language models, with three main results. (1) We develop a compute-efficient from-scratch pipeline that reduces the training budget from 7.7T tokens in Ouro to 310B tokens while retaining strong reasoning performance. Pretraining followed by high-quality mid-training, together with learning-rate warmup and stronger exit-gate regularization, enables stable recurrent training without prior multi-stage schedules. (2) Under controlled comparisons, our 1.4B LoopLM outperforms a parameter-matched dense model trained on the same data and token budget on all 12 evaluated benchmarks, including +14 points on GSM8K, +10 on MATH, and +22 on DROP. At matched inference compute, it approaches a 3.9B dense model on mathematical reasoning and reading comprehension while using only 36\% as many parameters. (3) We introduce a minimal recipe for converting pretrained dense models into looped ones: a single learned input-mixing scalar and a smoothed exit loss, with no step-specific parameters. Applied to Qwen3-1.7B-Base, Looped Qwen improves over an identically continued dense baseline on every evaluated benchmark across two data regimes, with statistically clear gains on GSM8K, MATH, and MMLU-Pro on the curated mixture. Together, these results make looped language models substantially cheaper to train from scratch and practical to introduce into existing pretrained checkpoints, while isolating the gains due to recurrence itself.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.00673 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.099Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).