Metraipolis › archive › dc_67e2ede8bd9c

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms — arXiv:2608.04074

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.04074
License (as archived)
CC0-1.0
Archived
2026-10-31T11:13:32.150Z
SHA-256 of the archived text
cb6bd8d07879b650e411423e0a10928b2180026cb60104607eef48c00608e555
Stable URL of this page
https://metraipolis.com/documento/dc_67e2ede8bd9c

How to cite

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms — arXiv:2608.04074 — arXiv (en), https://arxiv.org/abs/2608.04074 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_67e2ede8bd9c (Metraipolis archive dc_67e2ede8bd9c, sha256 cb6bd8d07879b650e411423e0a10928b2180026cb60104607eef48c00608e555)
BibTeX
@misc{dc67e2ede8bd9c,
  title = {Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms — arXiv:2608.04074},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04074},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_67e2ede8bd9c; sha256 cb6bd8d07879b650e411423e0a10928b2180026cb60104607eef48c00608e555},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04074. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Information Theory (cs.IT); Signal Processing (eess.SP).
Pagina del lavoro: https://arxiv.org/abs/2608.04074 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:02:13.477Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).