Metraipolis › archive › dc_eb930fb75029

Looped Transformers as Optimizers — arXiv:2609.37379

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.37379
License (as archived): CC0-1.0
Archived: 2026-10-07T17:31:03.216Z
SHA-256 of the archived text: 91721f2fa7c5702f4982ba45cf7a9a46988ca26f3f2068cec71f74a2d721acf0
Stable URL of this page: https://metraipolis.com/documento/dc_eb930fb75029

How to cite

Looped Transformers as Optimizers — arXiv:2609.37379 — arXiv (en), https://arxiv.org/abs/2609.37379 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb930fb75029 (Metraipolis archive dc_eb930fb75029, sha256 91721f2fa7c5702f4982ba45cf7a9a46988ca26f3f2068cec71f74a2d721acf0)

BibTeX

@misc{dceb930fb75029,
  title = {Looped Transformers as Optimizers — arXiv:2609.37379},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37379},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb930fb75029; sha256 91721f2fa7c5702f4982ba45cf7a9a46988ca26f3f2068cec71f74a2d721acf0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37379. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Looped Transformers provide a parameter-efficient approach to depth scaling by repeatedly applying shared Transformer blocks. Recent reasoning models have likewise highlighted the value of scaling test-time computation through longer computation trajectories. However, the principles for designing effective loop transitions remain poorly understood. We view the looped hidden state as a fast weight that is updated throughout the depth. We formulate loop transitions as local gradient-based updates, with recurrent blocks predicting implicit targets at each depth. Our framework derives loop transitions in closed form from a projection, a local objective and an optimizer update rule. Mapping representative loop transitions into this framework reveals mismatches between their transitions and projections. We first align the input maps of existing transitions. We then derive OperLoop, which combines explicit weight decay, adaptive step size and a delta objective. The aligned variants reduce training loss and improve average commonsense accuracy. OperLoop improves average generative performance over the compared looped and non-looped baselines under matched training FLOPs. These results support the framework's usefulness for loop design. We extend the analysis to additional loop models and outline a roadmap for future loop transition design.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.37379 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.317Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).