Metraipolis › archive › dc_67e2ede8bd9c
Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms — arXiv:2608.04074
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.04074
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T11:13:32.150Z
- SHA-256 of the archived text
cb6bd8d07879b650e411423e0a10928b2180026cb60104607eef48c00608e555- Stable URL of this page
https://metraipolis.com/documento/dc_67e2ede8bd9c
How to cite
Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms — arXiv:2608.04074 — arXiv (en), https://arxiv.org/abs/2608.04074 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_67e2ede8bd9c (Metraipolis archive dc_67e2ede8bd9c, sha256 cb6bd8d07879b650e411423e0a10928b2180026cb60104607eef48c00608e555)
BibTeX
@misc{dc67e2ede8bd9c,
title = {Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms — arXiv:2608.04074},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.04074},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_67e2ede8bd9c; sha256 cb6bd8d07879b650e411423e0a10928b2180026cb60104607eef48c00608e555},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04074. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long-context LLM decoding reads the key-value (KV) cache at every step. Loading it takes longer than computing attention over it, so throughput is bandwidth-bound. Hence, reducing the cache size can raise both decoding speed and serving capacity. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per-token bit count. At two bits per element, the most competitive methods rely on orthogonal transforms. However, existing techniques are either data-oblivious or use the query statistics without deriving the transform from a distortion criterion. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed-width scalar quantizers, which are suboptimal at low rates. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products. We derive closed-form optimal transforms for keys and values from calibration statistics, under a high-resolution model. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention-aware distortion becomes mean-squared error (MSE) in the transform domain. Thus, we can use MSE-optimal vector quantizers applied directly to the transformed key coefficients. To meet the fixed-width layout requirement, we show that grouping coefficients into equal-volume partitions makes equal-size codebooks attain the variable-rate optimum under the same high-resolution model. At two bits per element, our method, termed NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Information Theory (cs.IT); Signal Processing (eess.SP). Pagina del lavoro: https://arxiv.org/abs/2608.04074 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T11:02:13.477Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).