Metraipolis › archive › dc_62304fc2c83d

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers — arXiv:2609.01343

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01343
License (as archived)
CC0-1.0
Archived
2026-10-17T06:52:22.701Z
SHA-256 of the archived text
3def1bf5a5cb9cb024e2ecf879da6bcb5037fdb5bf9e427ff7c83b818bd734b6
Stable URL of this page
https://metraipolis.com/documento/dc_62304fc2c83d

How to cite

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers — arXiv:2609.01343 — arXiv (en), https://arxiv.org/abs/2609.01343 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_62304fc2c83d (Metraipolis archive dc_62304fc2c83d, sha256 3def1bf5a5cb9cb024e2ecf879da6bcb5037fdb5bf9e427ff7c83b818bd734b6)
BibTeX
@misc{dc62304fc2c83d,
  title = {SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers — arXiv:2609.01343},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01343},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_62304fc2c83d; sha256 3def1bf5a5cb9cb024e2ecf879da6bcb5037fdb5bf9e427ff7c83b818bd734b6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01343. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0\% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.01343 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:41:55.854Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).