Metraipolis › archive › dc_aa152b2e459f

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41850
License (as archived): CC0-1.0
Archived: 2026-11-03T07:41:43.262Z
SHA-256 of the archived text: 1a43a0fa17bd90eec77b488eff8e53fe57a3ef73544e22faeaddf1abda674267
Stable URL of this page: https://metraipolis.com/documento/dc_aa152b2e459f

How to cite

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850 — arXiv (en), https://arxiv.org/abs/2610.41850 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aa152b2e459f (Metraipolis archive dc_aa152b2e459f, sha256 1a43a0fa17bd90eec77b488eff8e53fe57a3ef73544e22faeaddf1abda674267)

BibTeX

@misc{dcaa152b2e459f,
  title = {Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41850},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aa152b2e459f; sha256 1a43a0fa17bd90eec77b488eff8e53fe57a3ef73544e22faeaddf1abda674267},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41850. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers retain an explicit history of the sequence, allowing attention to directly retrieve past representations. However, the size of the KV-Cache and the cost of a new token increase with the context length. State Space Models compress the past into a fixed-size recurrent state, at the cost of reduced addressability. We present the Dynamical Memory Transformer (DMT), a hybrid architecture that separates sequence encoding from information reading. The encoder uses several memory units, each implemented by a dynamical system with a complex diagonal recurrent matrix. An input-dependent leak rate controls their updates. The decoder then uses attention to read and combine their outputs, without going through the token history. The formulation of the recurrence allows an associative scan with O(T ) total work and O(log T ) parallel depth, with fixed dimensions. During autoregressive inference, the recurrent states and the convolution cache remain fixed in size. We compare DMT with Mamba and Transformer on cognitive and language modeling tasks, and show its strengths and limitations: DMT achieves the best results on several cognitive tasks and language modeling performance close to that of Transformer, but remains behind Mamba in language modeling. On selective copy, DMT also shows stronger length extrapolation than Mamba and Transformer. These results demonstrate that parallel dynamical memory units are a promising mechanism for retaining and using information over sequences.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41850 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:39:37.313Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).