Metraipolis › archive › dc_7404f954798a

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs — arXiv:2609.11744

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.11744
License (as archived)
CC0-1.0
Archived
2026-10-16T05:44:51.788Z
SHA-256 of the archived text
308a7e16f56360554557c05f701a55a39b93f3c9dd16fd61a462b145a4a07e98
Stable URL of this page
https://metraipolis.com/documento/dc_7404f954798a

How to cite

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs — arXiv:2609.11744 — arXiv (en), https://arxiv.org/abs/2609.11744 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7404f954798a (Metraipolis archive dc_7404f954798a, sha256 308a7e16f56360554557c05f701a55a39b93f3c9dd16fd61a462b145a4a07e98)
BibTeX
@misc{dc7404f954798a,
  title = {Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs — arXiv:2609.11744},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.11744},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7404f954798a; sha256 308a7e16f56360554557c05f701a55a39b93f3c9dd16fd61a462b145a4a07e98},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.11744. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: this https URL .

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-11 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.11744 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T05:26:44.601Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).