Metraipolis › archive › dc_e24d00a728b0

Trading Depth for Time in Recurrent Transformers — arXiv:2609.21605

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.21605
License (as archived): CC0-1.0
Archived: 2026-10-10T01:45:09.179Z
SHA-256 of the archived text: b7e964958d2f816c64cddc72d436201a27d157d41c26889dee19490d33f226d2
Stable URL of this page: https://metraipolis.com/documento/dc_e24d00a728b0

How to cite

Trading Depth for Time in Recurrent Transformers — arXiv:2609.21605 — arXiv (en), https://arxiv.org/abs/2609.21605 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e24d00a728b0 (Metraipolis archive dc_e24d00a728b0, sha256 b7e964958d2f816c64cddc72d436201a27d157d41c26889dee19490d33f226d2)

BibTeX

@misc{dce24d00a728b0,
  title = {Trading Depth for Time in Recurrent Transformers — arXiv:2609.21605},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21605},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e24d00a728b0; sha256 b7e964958d2f816c64cddc72d436201a27d157d41c26889dee19490d33f226d2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21605. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recurrent Transformers increase computational depth through temporal recurrence, feeding each token's high-level hidden state into the computation of the next. This raises a natural question: is additional computation better spent on more temporal steps or greater physical depth? We investigate this question using Latent Recurrent Transformers (LRTs), which retain one backbone forward pass per vocabulary token during decoding and provide a controlled setting for comparing these two ways of adding computation. Specifically, we insert a latent thought token between consecutive vocabulary tokens. Each thought token passes through the same $L$ layers as a vocabulary token, sharing the backbone parameters and providing an additional stage of hidden-state refinement before predicting the next token. We compare this $L$-layer LRT against a $2L$-layer LRT without thought tokens. Both execute $2L$ Transformer blocks per vocabulary token during decoding, but the thought-token model uses fewer parameters. On 16- and 20-layer mixture-of-experts NanoChat backbones, one thought token brings the shallower model within 0.006 and 0.004 bits per byte of its double-depth counterpart, recovering 67% and 81% of the improvement with approximately 48% fewer total parameters. These results suggest that temporal thinking offers a parameter-efficient alternative to increasing physical depth in recurrent Transformers.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.21605 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:14:16.639Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).