Metraipolis › archive › dc_f7ce2a2adcc8

Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation — arXiv:2609.04263

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04263
License (as archived)
CC0-1.0
Archived
2026-10-16T22:35:46.670Z
SHA-256 of the archived text
1c368b12e6d601639eb5df817ab64c495d6e096abc65880e4800e713d3f00b73
Stable URL of this page
https://metraipolis.com/documento/dc_f7ce2a2adcc8

How to cite

Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation — arXiv:2609.04263 — arXiv (en), https://arxiv.org/abs/2609.04263 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f7ce2a2adcc8 (Metraipolis archive dc_f7ce2a2adcc8, sha256 1c368b12e6d601639eb5df817ab64c495d6e096abc65880e4800e713d3f00b73)
BibTeX
@misc{dcf7ce2a2adcc8,
  title = {Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation — arXiv:2609.04263},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04263},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f7ce2a2adcc8; sha256 1c368b12e6d601639eb5df817ab64c495d6e096abc65880e4800e713d3f00b73},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04263. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-bit key--value (KV) caches reduce the memory required for autoregressive decoding, but the resulting quality loss depends on the model and quantizer. We keep the quantizer fixed and distill the floating-cache model's behavior into low-rank Q/K/V projection updates while the student executes a physically packed incremental cache. Across three seeds, 4-bit affine-cache adapters recover $54.24\%\pm2.47\%$ of the held-out perplexity gap on TinyLlama-1.1B and $75.96\%\pm4.04\%$ on Gemma-4-12B. On the same frozen NF4 Llama-3.1-8B base, one validation-selected run per quantizer recovers $60.42\%$ under KIVI K2V2 and $37.61\%$ under KVarN K4V2, while preserving 180-case associative retrieval. Gemma's score on an official 4K/8K RULER subset rises from 42.80 with the unadapted 4-bit cache to 48.33 after adaptation (46.15 floating), with substantial task heterogeneity. Finally, a 2-bit rank--token sweep reduces TinyLlama's 2-bit PPL from 576.10 to $11.4000\pm0.0059$ across three seeds, versus 10.3988 floating, but restores only 11--12 of 180 retrieval cases. These results show that low-rank projection adaptation can recover held-out quality across fixed cache formats, while perplexity recovery need not restore long-context retrieval.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.CL).
Categorie arXiv: Hardware Architecture (cs.AR) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.04263 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:21:16.310Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).