Metraipolis › archive › dc_798c28833773

Memory Attention — arXiv:2609.28399

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.28399
License (as archived)
CC0-1.0
Archived
2026-10-08T10:57:40.651Z
SHA-256 of the archived text
15253768b8d144e52f1c541bb1505c26ccb7e1b07ebe6c5bf2183c7745079f1c
Stable URL of this page
https://metraipolis.com/documento/dc_798c28833773

How to cite

Memory Attention — arXiv:2609.28399 — arXiv (en), https://arxiv.org/abs/2609.28399 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_798c28833773 (Metraipolis archive dc_798c28833773, sha256 15253768b8d144e52f1c541bb1505c26ccb7e1b07ebe6c5bf2183c7745079f1c)
BibTeX
@misc{dc798c28833773,
  title = {Memory Attention — arXiv:2609.28399},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.28399},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_798c28833773; sha256 15253768b8d144e52f1c541bb1505c26ccb7e1b07ebe6c5bf2183c7745079f1c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.28399. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and addition. Token-indexed retrieval also enables CPU offloading with prefetching, reducing GPU parameter storage. Under matched training token budgets and with additional memory parameters, experiments across attention configurations show improved language modeling and average downstream performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.28399 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:06:16.780Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).