Metraipolis › archive › dc_4cf35deb9343
Maglev: Sliding Recurrent Memory — arXiv:2608.02870
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.02870
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:42:15.864Z
- SHA-256 of the archived text
a0b4fb35f48d57c7371d4329cbdad1b77cba9af46562ad42788ab78974e937b1- Stable URL of this page
https://metraipolis.com/documento/dc_4cf35deb9343
How to cite
Maglev: Sliding Recurrent Memory — arXiv:2608.02870 — arXiv (en), https://arxiv.org/abs/2608.02870 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4cf35deb9343 (Metraipolis archive dc_4cf35deb9343, sha256 a0b4fb35f48d57c7371d4329cbdad1b77cba9af46562ad42788ab78974e937b1)
BibTeX
@misc{dc4cf35deb9343,
title = {Maglev: Sliding Recurrent Memory — arXiv:2608.02870},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.02870},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4cf35deb9343; sha256 a0b4fb35f48d57c7371d4329cbdad1b77cba9af46562ad42788ab78974e937b1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02870. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. \ours{} consists of two coupled models: a prefiller $Q$, which leverages full attention\footnote{In practice, we use interleaved full and sliding-window attention for $Q$, as this yields stronger performance. The essential requirement is that $Q$ be more expressive than $P$, with access to the full history.} to produce memory targets $m'_t$, and a decoder $P$, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories $m_t$ for next-token prediction. We train \ours{} with a memory consistency loss that aligns $m_t$ with $m'_t$, allowing inference to use $P$ alone. Empirically, \ours{} improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between $P$ and $Q$ reduces parameter memory while preserving most of the gains.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.02870 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:31:30.323Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).