Metraipolis › archive › dc_3f4695a2ecb4
KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference — arXiv:2608.21362
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.21362
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T07:38:34.547Z
- SHA-256 of the archived text
0113babd930e9011be9a0150c99678c8e7765ec53c4a67082e4b92a8c3525615- Stable URL of this page
https://metraipolis.com/documento/dc_3f4695a2ecb4
How to cite
KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference — arXiv:2608.21362 — arXiv (en), https://arxiv.org/abs/2608.21362 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3f4695a2ecb4 (Metraipolis archive dc_3f4695a2ecb4, sha256 0113babd930e9011be9a0150c99678c8e7765ec53c4a67082e4b92a8c3525615)
BibTeX
@misc{dc3f4695a2ecb4,
title = {KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference — arXiv:2608.21362},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.21362},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3f4695a2ecb4; sha256 0113babd930e9011be9a0150c99678c8e7765ec53c4a67082e4b92a8c3525615},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.21362. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing prefix-caching systems reduce this cost but require prompts to share a leading contiguous prefix, limiting effectiveness when shared content appears at arbitrary positions. We present KVBoost, a chunk-level KV cache reuse system for HuggingFace-compatible decoder models that enables reuse regardless of content position. KVBoost introduces a dual-hash keying scheme that separates positional identity (prefix hash) from content identity (content hash), supporting both exact and approximate cache matches. To address attention boundary errors from independently cached chunks, KVBoost employs two repair strategies: SelectiveRecompute, which re-encodes boundary regions, and CacheBlendRecompute, which identifies and recomputes high-deviation tokens after a probe pass. The system further incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction under a fixed memory budget. Evaluated on Qwen/Qwen2.5-3B over 1,000 bug-localization samples, KVBoost achieves a 4.49x reduction in time-to-first-token (142.4 ms vs.\ 639.1 ms) and outperforms prefix caching by 16%, with no loss in accuracy (99.2% vs.\ 99.1%). KVBoost provides a practical, memory-bounded inference acceleration layer compatible with RoPE-based models without architectural modification. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-25 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2608.21362 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T07:37:52.624Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).