Metraipolis › archive › dc_adf80968ac00
Memory as a cache: Exact context reuse and deletion by construction — arXiv:2609.32395
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32395
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:55:23.246Z
- SHA-256 of the archived text
69b2ead5c9cd60eede2d3d11cb765845a3b4553782f3ab1d2effb4315636b879- Stable URL of this page
https://metraipolis.com/documento/dc_adf80968ac00
How to cite
Memory as a cache: Exact context reuse and deletion by construction — arXiv:2609.32395 — arXiv (en), https://arxiv.org/abs/2609.32395 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_adf80968ac00 (Metraipolis archive dc_adf80968ac00, sha256 69b2ead5c9cd60eede2d3d11cb765845a3b4553782f3ab1d2effb4315636b879)
BibTeX
@misc{dcadf80968ac00,
title = {Memory as a cache: Exact context reuse and deletion by construction — arXiv:2609.32395},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32395},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_adf80968ac00; sha256 69b2ead5c9cd60eede2d3d11cb765845a3b4553782f3ab1d2effb4315636b879},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32395. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The KV cache of a transformer entangles every token's representation with its entire prefix: a passage encoded once cannot be reused under a different prefix or removed without recomputing everything after it, so exact cache reuse is limited to shared prefixes. We present SMem, an architecture whose context representation is a cache by construction. A block-local encoder maps each block to memory rows independently of other blocks, and a reader conditions generation on their union through cross-attention. For every parameter setting, memory composes exactly at fixed block indices, deleting a block is an exact $O(b)$ update for $b$-token blocks, and the memory state is independent of the edit path. At $4\times$ the training context, under the shared recipe, SMem retrieves planted needles beyond any trained-length window (exact match 0.14-0.28 at distances of 31 and 63 blocks), where learned-position, RoPE, and Block-Attention-style transformers all score at most 0.02. A fully cached context is served by computing one block alone at a near-constant 3.1-6.2 ms, whereas cold prefill grows with context; batched decode stores 34-38% fewer KV rows and runs 1.4-1.7$\times$ faster when bandwidth-bound; and deletion beats suffix recomputation by 8.5$\times$ at 512 blocks and 452$\times$ at 4096 blocks (32-256$\times$ the trained length, probing the cost model rather than a served regime). The cost is a perplexity gap of -4.7% to +2.8% (negative favors SMem) against a parameter-matched transformer with the same positional scheme, at 160M-1.5B on FineWeb-Edu across two recipes and a learning-rate search. SMem also composes with RoPE: at 160M and 410M the composite matches or leads the matched transformer and closes 29-59% of SMem's gap to a RoPE transformer. Dropping prefix entanglement thus keeps perplexity comparable while making the cache exactly composable and editable. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.32395 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:00.780Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).