Metraipolis › archive › dc_2ef4197a9345
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models — arXiv:2609.02029
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.02029
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T03:25:46.354Z
- SHA-256 of the archived text
f8685cde6fe4e39614df3da216dcadaa489ac2e61975b83e34d46ddd05d557f7- Stable URL of this page
https://metraipolis.com/documento/dc_2ef4197a9345
How to cite
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models — arXiv:2609.02029 — arXiv (en), https://arxiv.org/abs/2609.02029 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2ef4197a9345 (Metraipolis archive dc_2ef4197a9345, sha256 f8685cde6fe4e39614df3da216dcadaa489ac2e61975b83e34d46ddd05d557f7)
BibTeX
@misc{dc2ef4197a9345,
title = {HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models — arXiv:2609.02029},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.02029},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2ef4197a9345; sha256 f8685cde6fe4e39614df3da216dcadaa489ac2e61975b83e34d46ddd05d557f7},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02029. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long-context inference retains a growing key--value (KV) cache during decoding, which consumes substantial GPU memory and can reduce generation throughput. This bottleneck remains in hybrid language models because their residual global-attention layers can dominate context-dependent cache demand. We study how to allocate this state under an aggregate KV-residency budget. We introduce HeadWiseKV, a training-free framework that compresses the residual global KV caches of hybrid language models while preserving their native local, recurrent, and linear paths. It assigns each physical KV head a static, multilevel history window, making cache demand predictable before serving. We formulate this allocation as a restricted operational rate--distortion problem and propose SeqCalib as the core policy-generation algorithm in HeadWiseKV. SeqCalib processes layers in execution order and conditions each decision on the lower-layer policy used at deployment, thereby accounting for interactions across depth. A grouped-cache runtime materializes the selected policy as actual per-head KV residency rather than a mask over a full cache. We evaluate downstream quality across four hybrid long-context models and study physical residency and serving behavior on Qwen3.6-27B. HeadWiseKV retains near-Full-KV RULER and LoCoMo quality across the evaluated models. In the fixed-model systems study, it reduces sampled peak device memory by 8.59\% at a 112K context length and extends the largest verified successful context from 114K to 161K. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.02029 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T03:13:12.412Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).