Metraipolis › archive › dc_487c41fc9def

Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference — arXiv:2609.32663

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32663
License (as archived)
CC0-1.0
Archived
2026-10-07T21:53:57.466Z
SHA-256 of the archived text
7db7e85651d0e16cb2b790f74a3d8873d8de6ab9326a163accc900d3c268ef6b
Stable URL of this page
https://metraipolis.com/documento/dc_487c41fc9def

How to cite

Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference — arXiv:2609.32663 — arXiv (en), https://arxiv.org/abs/2609.32663 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_487c41fc9def (Metraipolis archive dc_487c41fc9def, sha256 7db7e85651d0e16cb2b790f74a3d8873d8de6ab9326a163accc900d3c268ef6b)
BibTeX
@misc{dc487c41fc9def,
  title = {Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference — arXiv:2609.32663},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32663},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_487c41fc9def; sha256 7db7e85651d0e16cb2b790f74a3d8873d8de6ab9326a163accc900d3c268ef6b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32663. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The memory usage and decoding latency of LLM inference grow rapidly with context length. To reduce these costs, key-value (KV) cache compression methods selectively retain cached states based on token importance or differences in attention patterns across heads. However, we discover that retrieval capability varies substantially with relative distance, even within the same attention head. To exploit this structure, we introduce Distance-KV, which learns a static KV retention pattern over the joint space of layers, attention heads, and relative distances. The pattern is learned offline with the language model frozen and reused across inputs to prune and compact the KV cache without online importance scoring. Across three backbone models and four long-context benchmarks, Distance-KV consistently achieves the best overall performance among competing KV cache compression methods, exceeding the strongest compression baseline by up to 9.3 points on RULER at 128K. On Llama-3.1-8B-Instruct at 128K, Distance-KV reduces KV cache memory by 65.4% and achieves a $1.66\times$ decoding speedup relative to Dense. Together, these results identify relative distance as an important structural dimension for understanding how LLMs retrieve information over long contexts and for designing more efficient inference methods.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.32663 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.742Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).