Metraipolis › archive › dc_42ed9f024a88

Causal Latent Workspaces for Looped Transformers — arXiv:2610.22796

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22796
License (as archived)
CC0-1.0
Archived
2026-10-19T08:43:35.316Z
SHA-256 of the archived text
91efd6f2090c47049c975ced8c1efa2bdee46b047a5a83d0006f3aa11bbfa16b
Stable URL of this page
https://metraipolis.com/documento/dc_42ed9f024a88

How to cite

Causal Latent Workspaces for Looped Transformers — arXiv:2610.22796 — arXiv (en), https://arxiv.org/abs/2610.22796 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_42ed9f024a88 (Metraipolis archive dc_42ed9f024a88, sha256 91efd6f2090c47049c975ced8c1efa2bdee46b047a5a83d0006f3aa11bbfa16b)
BibTeX
@misc{dc42ed9f024a88,
  title = {Causal Latent Workspaces for Looped Transformers — arXiv:2610.22796},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22796},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_42ed9f024a88; sha256 91efd6f2090c47049c975ced8c1efa2bdee46b047a5a83d0006f3aa11bbfa16b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22796. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reusing a Transformer block across depth increases computation without replicating its parameters, but successive passes have no dedicated state through which positions can exchange compact summaries of their causal prefix. We introduce the causal latent workspace Transformer: an anchored, four-round shared-depth architecture in which each position writes gated vectors into prefix-specific latent slots and retrieves a content-addressed summary before the next shared block application. This path complements causal self-attention while the original input embedding remains accessible at every round. Across three byte-level corpora, three seeds, and two matched parameter budgets, the proposed model achieves the best mean held-out perplexity in five of six settings against five Transformer controls. At 101.71M parameters it leads on WikiText-103 raw, enwik8, and TinyStories, reaching perplexities 2.7820, 2.7843, and 1.6823; these improve upon the strongest non-workspace control on each corpus by 1.06%, 0.73%, and 1.01%. All nine seed-labelled comparisons at this budget favor the workspace. At 0.918M parameters it leads on WikiText-103 and TinyStories, whereas a conditioned loop is stronger on enwik8. Architectural ablations show that adding the workspace improves upon the anchored loop in all six settings and support refreshing its causal summaries at each round. Together, these results identify a causal, content-addressable prefix workspace as an effective structural augmentation of looped Transformers.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.22796 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:24.832Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).