Metraipolis › archive › dc_582d3395ddbc

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching — arXiv:2608.08097

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.08097
License (as archived)
CC0-1.0
Archived
2026-10-19T08:52:35.523Z
SHA-256 of the archived text
17f9cf13ed6cb4f66f4134711628805fdd8faeb3f88878fe0efa2ada066bbbc6
Stable URL of this page
https://metraipolis.com/documento/dc_582d3395ddbc

How to cite

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching — arXiv:2608.08097 — arXiv (en), https://arxiv.org/abs/2608.08097 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_582d3395ddbc (Metraipolis archive dc_582d3395ddbc, sha256 17f9cf13ed6cb4f66f4134711628805fdd8faeb3f88878fe0efa2ada066bbbc6)
BibTeX
@misc{dc582d3395ddbc,
  title = {OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching — arXiv:2608.08097},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.08097},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_582d3395ddbc; sha256 17f9cf13ed6cb4f66f4134711628805fdd8faeb3f88878fe0efa2ada066bbbc6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.08097. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: $1.69\times$ over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to $2.1\times$ on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about $2\times$ dense throughput while admitting each request with $6.5$--$9.7\times$ less KV and holding $2.2$-$2.6$ less decode-node host memory than full KV transfer.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-11 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.08097 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:50:23.730Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).