Metraipolis › archive › dc_f7ea5659cbad

Read What Matters: Query-Adaptive Quantization for KV Caches — arXiv:2610.11245

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11245
License (as archived)
CC0-1.0
Archived
2026-10-09T06:13:47.167Z
SHA-256 of the archived text
4c8341ab345f45476ae2b369fc4bd4a14433d1737aeb53d8480c53063368e3bb
Stable URL of this page
https://metraipolis.com/documento/dc_f7ea5659cbad

How to cite

Read What Matters: Query-Adaptive Quantization for KV Caches — arXiv:2610.11245 — arXiv (en), https://arxiv.org/abs/2610.11245 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f7ea5659cbad (Metraipolis archive dc_f7ea5659cbad, sha256 4c8341ab345f45476ae2b369fc4bd4a14433d1737aeb53d8480c53063368e3bb)
BibTeX
@misc{dcf7ea5659cbad,
  title = {Read What Matters: Query-Adaptive Quantization for KV Caches — arXiv:2610.11245},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11245},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f7ea5659cbad; sha256 4c8341ab345f45476ae2b369fc4bd4a14433d1737aeb53d8480c53063368e3bb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11245. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV stores each key and value in a progressive code whose prefixes support different reconstruction precisions. For each query, it allocates key-channel prefixes using the query, computes attention from the reconstructed keys, and then allocates value-token prefixes using that attention. Stored entries remain unchanged. Each stage optimizes a calibrated distortion objective under a fixed budget; we prove exact allocation under diminishing refinement gains and relate these objectives to attention-output error. We also exhibit a finite-dimensional attention family where query-dependent access strictly outperforms every query-independent reader at the same read budget, even with unrestricted competing encoders and decoders. Across six base models, reading four bits on average from an eight-bit cache increases C4 perplexity by at most 0.66%, using about one quarter of the logical reads and half the retained capacity of a 16-bit cache. It is consistently more accurate than storing and fully reading four bits at the same payload-read budget. Retaining more bits than each query fetches is aimed at long-context decoding, where the cache bytes moved per step, rather than the weights, dominate cost. Long-context question answering and retrieval on two instruction-tuned models provide additional quality evidence. On the tested 8K-token, batch-one, single-layer workload on an NVIDIA A10G, a restricted eight-bit ReadKV reader with a two-bit mean payload-read budget has 39% lower latency than the tested TurboQuant codec.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2610.11245 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:15.490Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).