Metraipolis › archive › dc_4cf35deb9343

Maglev: Sliding Recurrent Memory — arXiv:2608.02870

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.02870
License (as archived)
CC0-1.0
Archived
2026-10-31T12:42:15.864Z
SHA-256 of the archived text
a0b4fb35f48d57c7371d4329cbdad1b77cba9af46562ad42788ab78974e937b1
Stable URL of this page
https://metraipolis.com/documento/dc_4cf35deb9343

How to cite

Maglev: Sliding Recurrent Memory — arXiv:2608.02870 — arXiv (en), https://arxiv.org/abs/2608.02870 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4cf35deb9343 (Metraipolis archive dc_4cf35deb9343, sha256 a0b4fb35f48d57c7371d4329cbdad1b77cba9af46562ad42788ab78974e937b1)
BibTeX
@misc{dc4cf35deb9343,
  title = {Maglev: Sliding Recurrent Memory — arXiv:2608.02870},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.02870},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4cf35deb9343; sha256 a0b4fb35f48d57c7371d4329cbdad1b77cba9af46562ad42788ab78974e937b1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02870. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. \ours{} consists of two coupled models: a prefiller $Q$, which leverages full attention\footnote{In practice, we use interleaved full and sliding-window attention for $Q$, as this yields stronger performance. The essential requirement is that $Q$ be more expressive than $P$, with access to the full history.} to produce memory targets $m'_t$, and a decoder $P$, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories $m_t$ for next-token prediction. We train \ours{} with a memory consistency loss that aligns $m_t$ with $m'_t$, allowing inference to use $P$ alone. Empirically, \ours{} improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between $P$ and $Q$ reduces parameter memory while preserving most of the gains.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.02870 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:31:30.323Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).