Metraipolis › archive › dc_4e05ea18b49f

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts — arXiv:2610.01153

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.01153
License (as archived): CC0-1.0
Archived: 2026-10-06T12:12:58.565Z
SHA-256 of the archived text: 006b884428002a0a07d2e78d2c4e59961be033dd15d2b597dc25fd9826ac0ea8
Stable URL of this page: https://metraipolis.com/documento/dc_4e05ea18b49f

How to cite

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts — arXiv:2610.01153 — arXiv (en), https://arxiv.org/abs/2610.01153 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e05ea18b49f (Metraipolis archive dc_4e05ea18b49f, sha256 006b884428002a0a07d2e78d2c4e59961be033dd15d2b597dc25fd9826ac0ea8)

BibTeX

@misc{dc4e05ea18b49f,
  title = {Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts — arXiv:2610.01153},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01153},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e05ea18b49f; sha256 006b884428002a0a07d2e78d2c4e59961be033dd15d2b597dc25fd9826ac0ea8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01153. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Looped Transformers introduce recurrent depth as a new scaling axis for LLMs: by repeatedly applying shared Transformer blocks, they increase effective depth without increasing parameter count. However, the benefits of looping remain unclear for large MoE LLMs under FLOPs-matched comparisons. The main reason is that the gains from additional iterations diminish quickly and can even turn into degradation, so the extra FLOPs spent on looping yield little substantial improvement. Consequently, prior work typically settles on two loops. We identify two main obstacles to scaling looped MoE. First, looping inherits and amplifies the curse of depth: hidden-state variance grows with each iteration as residual updates accumulate, which destabilizes deep recurrence and causes representations to drift. Second, looped MoE suffers from expert selection collapse: routers repeatedly select the same experts across loops, so extra iterations add computation without adding computational diversity. Guided by this diagnosis, we propose LOOM, built on a single principle: each loop should contribute new computation while keeping the recurrent state stable. LOOM stabilizes recurrence by scaling residual updates to bound variance growth and re-injecting the input embedding at every loop, and diversifies it through per-loop routers that engage different experts and a Looping Residual that carries earlier outputs forward. Experiments across 100M-1.7B models show stable scaling to 9-12 loops. Under near-iso-FLOP, the 700M model performs best at 5 loops, reducing perplexity from 18.36 to 16.54 and improving average zero-shot accuracy from 38.84% to 39.53% over the non-looped baseline. Without FLOP matching, the 1.7B model trained on 60B tokens peaks at 9 loops, reducing perplexity from 9.62 to 7.77 and improving average zero-shot accuracy from 42.4% to 47.7%. Code is available this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.01153 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.159Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).