Metraipolis › archive › dc_abfd66c8ed0d
How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference — arXiv:2610.40325
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.40325
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:14:52.554Z
- SHA-256 of the archived text
d77b5f0dfa3b6a3a8afa919b208d7325eef786feaf6f4d86da7f6a93b3de5c28- Stable URL of this page
https://metraipolis.com/documento/dc_abfd66c8ed0d
How to cite
How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference — arXiv:2610.40325 — arXiv (en), https://arxiv.org/abs/2610.40325 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_abfd66c8ed0d (Metraipolis archive dc_abfd66c8ed0d, sha256 d77b5f0dfa3b6a3a8afa919b208d7325eef786feaf6f4d86da7f6a93b3de5c28)
BibTeX
@misc{dcabfd66c8ed0d,
title = {How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference — arXiv:2610.40325},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.40325},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_abfd66c8ed0d; sha256 d77b5f0dfa3b6a3a8afa919b208d7325eef786feaf6f4d86da7f6a93b3de5c28},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40325. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these trade-offs jointly in dense decoder-only transformers trained from scratch, varying which parts of the model are looped, how many loops are applied, and how information is passed between loops. Our experiments span models with up to 1B parameters, trained on up to 155B tokens with up to $1.07\times10^{21}$ FLOPs and 32 loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained. Adding loops is not uniformly beneficial: gains eventually saturate, reverse, or become unstable. This motivates asking whether forcing every loop to use the same weights limits the benefit of deeper looping. We address this by adding small loop-specific low-rank adapters that improve quality, with larger gains as more loops are added. Looped transformers also suffer a decoding slowdown proportional to the number of loops. Parallel Loop Transformers address this by passing hidden states between loops across adjacent token positions, allowing loop stages to run concurrently. However, in our experiments the original formulation does not consistently retain the quality gains of ordinary looping. We find that preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovers strong performance across model scales and training budgets. Together, these results show that effective looping requires matching the design to the resource constraint and inference setting.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.40325 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:08.766Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).