Metraipolis › archive › dc_ccac271d67a4
Learning Length-Extrapolatable Recurrent Models — arXiv:2609.09157
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.09157
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:58:45.345Z
- SHA-256 of the archived text
a4edc1001f794bd4d88c73baaa56d7a1603407e971c985b7f49c4fdad21d69a4- Stable URL of this page
https://metraipolis.com/documento/dc_ccac271d67a4
How to cite
Learning Length-Extrapolatable Recurrent Models — arXiv:2609.09157 — arXiv (en), https://arxiv.org/abs/2609.09157 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ccac271d67a4 (Metraipolis archive dc_ccac271d67a4, sha256 a4edc1001f794bd4d88c73baaa56d7a1603407e971c985b7f49c4fdad21d69a4)
BibTeX
@misc{dcccac271d67a4,
title = {Learning Length-Extrapolatable Recurrent Models — arXiv:2609.09157},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.09157},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ccac271d67a4; sha256 a4edc1001f794bd4d88c73baaa56d7a1603407e971c985b7f49c4fdad21d69a4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09157. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.09157 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:25.396Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).