Metraipolis › archive › dc_a776ff092cf3

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41850
License (as archived): CC0-1.0
Archived: 2026-11-02T07:29:15.255Z
SHA-256 of the archived text: 4035e2a13355814f9440a77fbdc2a39f3019be41d234536988245f02da9eaeb9
Stable URL of this page: https://metraipolis.com/documento/dc_a776ff092cf3

How to cite

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850 — arXiv (en), https://arxiv.org/abs/2610.41850 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a776ff092cf3 (Metraipolis archive dc_a776ff092cf3, sha256 4035e2a13355814f9440a77fbdc2a39f3019be41d234536988245f02da9eaeb9)

BibTeX

@misc{dca776ff092cf3,
  title = {Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention — arXiv:2610.41850},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41850},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a776ff092cf3; sha256 4035e2a13355814f9440a77fbdc2a39f3019be41d234536988245f02da9eaeb9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41850. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers retain an explicit history of the sequence, allowing attention to directly retrieve past representations. However, the size of the KV-Cache and the cost of a new token increase with the context length. State Space Models compress the past into a fixed-size recurrent state, at the cost of reduced addressability. We present the Dynamical Memory Transformer (DMT), a hybrid architecture that separates sequence encoding from information reading. The encoder uses several memory units, each implemented by a dynamical system with a complex diagonal recurrent matrix. An input-dependent leak rate controls their updates. The decoder then uses attention to read and combine their outputs, without going through the token history. The formulation of the recurrence allows an associative scan with O(T ) total work and O(log T ) parallel depth, with fixed dimensions. During autoregressive inference, the recurrent states and the convolution cache remain fixed in size. We compare DMT with Mamba and Transformer on cognitive and language modeling tasks, and show its strengths and limitations: DMT achieves the best results on several cognitive tasks and language modeling performance close to that of Transformer, but remains behind Mamba in language modeling. On selective copy, DMT also shows stronger length extrapolation than Mamba and Transformer. These results demonstrate that parallel dynamical memory units are a promising mechanism for retaining and using information over sequences.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41850 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:26:57.727Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).