Metraipolis › archive › dc_830ef57d0ae6
Pretraining Latent Information Feedback Transformers with Teacher Supervision — arXiv:2609.38149
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38149
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:57:18.688Z
- SHA-256 of the archived text
2cfe3c107f380ea29496c35c3736e3c027d8c3dc54aedfe33418ddcb7fa06475- Stable URL of this page
https://metraipolis.com/documento/dc_830ef57d0ae6
How to cite
Pretraining Latent Information Feedback Transformers with Teacher Supervision — arXiv:2609.38149 — arXiv (en), https://arxiv.org/abs/2609.38149 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_830ef57d0ae6 (Metraipolis archive dc_830ef57d0ae6, sha256 2cfe3c107f380ea29496c35c3736e3c027d8c3dc54aedfe33418ddcb7fa06475)
BibTeX
@misc{dc830ef57d0ae6,
title = {Pretraining Latent Information Feedback Transformers with Teacher Supervision — arXiv:2609.38149},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38149},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_830ef57d0ae6; sha256 2cfe3c107f380ea29496c35c3736e3c027d8c3dc54aedfe33418ddcb7fa06475},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38149. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative continuations. In this work, we remove this bottleneck during pretraining, introducing the LIFT (Latent Information Feedback Transformer) architecture and training method which enable LMs to propagate state across generation. We achieve this by turning recurrent-state learning into a teacher-forced prediction problem: each input token is paired with an information-dense state, derived from the next-token distribution of an off-the-shelf pretrained LM. The model, extended with a small number of additional parameters, is then trained to predict both the next token and the next state. As the input states are precomputed, pretraining remains fully parallel across positions. At inference, the model's own predicted states are fed back, with a minor computational overhead that decreases with model size. Experiments with pretrained models ranging from 135M to 1B parameters show that LIFT consistently outperforms standard Transformers and baselines on language modeling, downstream reasoning tasks, and procedural tasks under token-matched budget, while being on par with or ahead of compute-matched Transformers. Moreover, a controlled study on a state-tracking task shows that a tiny LIFT outperforms same-size Transformers trained on 8x more data, even when trained with the states of a Transformer that fails the task. Overall, we show that LMs can learn to exploit deep-to-shallow feedback during pretraining via scalable teacher supervision. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.38149 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:27:03.372Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).