Metraipolis › archive › dc_7d23ea5ec7d5
ARM: Attention with Routed-Memory for Learnable Sparse Control — arXiv:2609.24417
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.24417
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T07:07:44.476Z
- SHA-256 of the archived text
0ca6abf15b08aef1c9a256d0ecbc77e8cba4f9a434ab82c797bb01174f7cf5aa- Stable URL of this page
https://metraipolis.com/documento/dc_7d23ea5ec7d5
How to cite
ARM: Attention with Routed-Memory for Learnable Sparse Control — arXiv:2609.24417 — arXiv (en), https://arxiv.org/abs/2609.24417 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d23ea5ec7d5 (Metraipolis archive dc_7d23ea5ec7d5, sha256 0ca6abf15b08aef1c9a256d0ecbc77e8cba4f9a434ab82c797bb01174f7cf5aa)
BibTeX
@misc{dc7d23ea5ec7d5,
title = {ARM: Attention with Routed-Memory for Learnable Sparse Control — arXiv:2609.24417},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.24417},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d23ea5ec7d5; sha256 0ca6abf15b08aef1c9a256d0ecbc77e8cba4f9a434ab82c797bb01174f7cf5aa},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24417. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Despite advances in long-context inference, large language models (LLMs) remain fundamentally limited by the key-value (KV) caching mechanisms that are necessary for stable computation. Techniques such as selective token eviction and pruning have vastly mitigated these issues, but often discard core information to manage the growing cache. In this paper, we propose Attention with Routed Memory (ARM) a novel KV caching structure that introduces a fully differentiable, fixed-size memory system organized as a hierarchical router. Via a Gumbel-Softmax, ARM learns to select memory slots and perform sigmoid-gated updates that softly combine new and stored information, avoiding hard eviction and reducing information loss. By further training a policy to dynamically select varying amounts of memory at inference, ARM adapts its accesses for both simple contexts and inputs that require deeper reasoning, enabling more scalable and effective retrieval on both short- and long-contexts. Experimental results on standard commonsense and long-context reasoning benchmarks demonstrate that ARM achieves superior performance and efficiency compared to fixed KV-caching approaches, while remaining efficient and scalable in terms of both memory and generation latency. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.24417 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T06:58:09.414Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).