Metraipolis › archive › dc_4c005faf1671

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41850
License (as archived)
CC0-1.0
Archived
2026-11-02T07:39:37.186Z
SHA-256 of the archived text
b4ef83ac89d609b30bef3e959cec27f29615fa76d9aa618938d54a6fd5b790ea
Stable URL of this page
https://metraipolis.com/documento/dc_4c005faf1671

How to cite

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850 — arXiv (en), https://arxiv.org/abs/2610.41850 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4c005faf1671 (Metraipolis archive dc_4c005faf1671, sha256 b4ef83ac89d609b30bef3e959cec27f29615fa76d9aa618938d54a6fd5b790ea)
BibTeX
@misc{dc4c005faf1671,
  title = {Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41850},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4c005faf1671; sha256 b4ef83ac89d609b30bef3e959cec27f29615fa76d9aa618938d54a6fd5b790ea},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41850. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers retain an explicit history of the sequence, allowing attention to directly retrieve past representations. However, the size of the KV-Cache and the cost of a new token increase with the context length. State Space Models compress the past into a fixed-size recurrent state, at the cost of reduced addressability. We present the Dynamical Memory Transformer (DMT), a hybrid architecture that separates sequence encoding from information reading. The encoder uses several memory units, each implemented by a dynamical system with a complex diagonal recurrent matrix. An input-dependent leak rate controls their updates. The decoder then uses attention to read and combine their outputs, without going through the token history. The formulation of the recurrence allows an associative scan with O(T ) total work and O(log T ) parallel depth, with fixed dimensions. During autoregressive inference, the recurrent states and the convolution cache remain fixed in size. We compare DMT with Mamba and Transformer on cognitive and language modeling tasks, and show its strengths and limitations: DMT achieves the best results on several cognitive tasks and language modeling performance close to that of Transformer, but remains behind Mamba in language modeling. On selective copy, DMT also shows stronger length extrapolation than Mamba and Transformer. These results demonstrate that parallel dynamical memory units are a promising mechanism for retaining and using information over sequences.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41850 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:37:00.508Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).