Metraipolis › archive › dc_d1cc0fbdd98e

Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does — arXiv:2609.39892

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39892
License (as archived)
CC0-1.0
Archived
2026-10-07T13:46:47.717Z
SHA-256 of the archived text
a8020debfd28380aec45a6c6f393793816d4795b205654a509f837a60d384111
Stable URL of this page
https://metraipolis.com/documento/dc_d1cc0fbdd98e

How to cite

Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does — arXiv:2609.39892 — arXiv (en), https://arxiv.org/abs/2609.39892 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d1cc0fbdd98e (Metraipolis archive dc_d1cc0fbdd98e, sha256 a8020debfd28380aec45a6c6f393793816d4795b205654a509f837a60d384111)
BibTeX
@misc{dcd1cc0fbdd98e,
  title = {Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does — arXiv:2609.39892},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39892},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d1cc0fbdd98e; sha256 a8020debfd28380aec45a6c6f393793816d4795b205654a509f837a60d384111},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39892. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Looped Transformers repeatedly apply the same set of Transformer layers, giving them a recurrent architecture for latent computation. Their strong performance on iterative reasoning and length-generalization tasks suggests an appealing explanation: recurrence may provide an inductive bias that lets the model reuse a learned algorithm across loops. However, weight sharing alone does not imply that every loop performs the same operation. This raises a basic question: is each loop actually repeating the same computation, and if not, what routes the shared parameters to different operations? We study this question using graph walks as a test case. In the model's native trajectories, decoded predictions can advance by different numbers of graph steps or remain at a reached target, showing that recurrent progress need not follow a fixed one-loop-one-step pattern. We then show that a frozen loop can be steered toward different transitions by modifying its entering hidden state: a learned linear layer $J$ selects the desired transition without changing the shared Transformer layers. To test how this steering works, we use activation patching and find that attention patterns can recover its effects and switch the selected transition. Across five matched pairs of graph models, changing intermediate supervision during backbone training changes which transitions $J$ can induce. This suggests that $J$ selects computations learned by the backbone rather than creating new algorithms. Together, these results show that the hidden state can control shared computation, with attention routing as a causal pathway.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.39892 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T13:18:10.487Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).