Metraipolis › archive › dc_db0cf69894d8
Shared Global KV with Layer-Specific Local History — arXiv:2609.28006
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.28006
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T10:57:33.387Z
- SHA-256 of the archived text
cc198d317c0c4c8fa41fc46ceb42e267820e26dca65c2f74a8d27c4db28240c4- Stable URL of this page
https://metraipolis.com/documento/dc_db0cf69894d8
How to cite
Shared Global KV with Layer-Specific Local History — arXiv:2609.28006 — arXiv (en), https://arxiv.org/abs/2609.28006 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_db0cf69894d8 (Metraipolis archive dc_db0cf69894d8, sha256 cc198d317c0c4c8fa41fc46ceb42e267820e26dca65c2f74a8d27c4db28240c4)
BibTeX
@misc{dcdb0cf69894d8,
title = {Shared Global KV with Layer-Specific Local History — arXiv:2609.28006},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.28006},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_db0cf69894d8; sha256 cc198d317c0c4c8fa41fc46ceb42e267820e26dca65c2f74a8d27c4db28240c4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.28006. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Decoder-only Transformer language models cache keys and values (KV) to reuse past computation during generation. Sharing KV across layers saves storage but reduces the diversity of representations available across depth. We study what local memory should retain alongside shared global KV, separating historical content from the input source used to form it. At 126M parameters and 2K context, an eight-seed study finds about 1.4% lower held-out test perplexity with local history than with a current-token local branch. Capacity, entry-count and training-compute controls support the value of historical content. In a two-seed comparison, this value persists when adjacent layers share local inputs while retaining independent projections; source sharing also shortens exact cache-construction dependencies. Against GQA and adjacent-layer KV sharing, equal bounded learning-rate searches and new-seed confirmation yield better same-source likelihood with larger caches and higher long-request latency. The ordering against adjacent-layer sharing persists after equal-token adaptation to 8K, with a short-context cost. The eight-seed external-book history effect remains uncertain, and downstream outcomes vary by task. We derive a sufficient suffix schedule that reduces upper-layer construction work while preserving the complete cache in exact arithmetic. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.28006 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:06:16.768Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).