Metraipolis › archive › dc_c49da4017180
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers — arXiv:2608.31067
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.31067
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T08:39:32.240Z
- SHA-256 of the archived text
4b47f3a642800c91827718523cb941ddd8f3270af9f232d98a8bd6891899a607- Stable URL of this page
https://metraipolis.com/documento/dc_c49da4017180
How to cite
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers — arXiv:2608.31067 — arXiv (en), https://arxiv.org/abs/2608.31067 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c49da4017180 (Metraipolis archive dc_c49da4017180, sha256 4b47f3a642800c91827718523cb941ddd8f3270af9f232d98a8bd6891899a607)
BibTeX
@misc{dcc49da4017180,
title = {Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers — arXiv:2608.31067},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.31067},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c49da4017180; sha256 4b47f3a642800c91827718523cb941ddd8f3270af9f232d98a8bd6891899a607},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.31067. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length generalization benchmarks. We present a provably correct, transformer parameterization (with only 280 learnable parameters for Boolean algebra tasks) capable of learning and evaluating problems of any depth or length. We assume inputs are fully parenthesized, well-formed expressions. Our approach conceptualizes algorithmic tasks as circuit models embedded in transformers, enabling depth-1 circuit reduction in a single forward pass. To achieve depth generalization, we introduce a positional encoding that tracks each gate's depth within the circuit, enabling the model to identify evaluable subexpressions at each iteration via masked hard attention, with $O(n)$ per-iteration complexity via linear attention. Combined with an autonomous halting criterion, the model terminates after $d$ iterations for problems of depth $d$, yielding $O(n \cdot d)$ total complexity. We show that training on shallow problem instances (depth 1 and depth 2) effectively recovers interpretable parameters that {\em snap} into place, resulting in exact length generalization. Though we establish that our construction provably evaluates Boolean expressions -- a universal symbolic computation -- of arbitrary length perfectly, in other experiments we also demonstrate that our transformer variant can learn and generalize perfectly (100% accuracy) on other common length generalization benchmarks, including modular arithmetic and ListOps.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.31067 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T08:13:26.865Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).