Metraipolis › archive › dc_6f44229c91ec
KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving — arXiv:2610.31827
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.31827
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-26T19:54:36.214Z
- SHA-256 of the archived text
0eebe0a0173f7947c9cd59de10b151a071559ebccd3001dfb97bc67ff57f4192- Stable URL of this page
https://metraipolis.com/documento/dc_6f44229c91ec
How to cite
KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving — arXiv:2610.31827 — arXiv (en), https://arxiv.org/abs/2610.31827 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6f44229c91ec (Metraipolis archive dc_6f44229c91ec, sha256 0eebe0a0173f7947c9cd59de10b151a071559ebccd3001dfb97bc67ff57f4192)
BibTeX
@misc{dc6f44229c91ec,
title = {KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving — arXiv:2610.31827},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.31827},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6f44229c91ec; sha256 0eebe0a0173f7947c9cd59de10b151a071559ebccd3001dfb97bc67ff57f4192},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31827. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long-context LLM serving for agentic, coding, and document workloads reuses prefixes whose key-value (KV) cache outgrows GPU and host memory, putting storage I/O on the critical path of the inference step for retrieving prefixes. A single node-local NVMe absorbs all retrievals, while the parallel file system or cloud file store that nodes already mount carries almost no inference traffic. We cast prefix retrieval as aggregate-bandwidth scheduling across these tiers and present KVStream, a KV cache engine that batches a step's reads, defers writes behind the forward pass, replicates each chunk on every tier, and lets each tier adaptively claim the next unread chunks once it is free. Our performance model shows this routing attains the aggregate-bandwidth bound without prediction or tuning, whereas any static split misses it by a margin that grows as per-tier bandwidths drift or fluctuate. It also predicts that the time-to-first-token (TTFT) gain grows with load, as faster retrieval drains the queue, and disappears when host memory or arrival order keeps reuse in cache. Plugged into LMCache and evaluated on four models, six datasets, and up to 128 GPUs, KVStream retrieves within 5% of the bound, delivers up to 1.49$\times$ higher offline throughput than LMCache, and lowers mean TTFT by up to 6.1$\times$ under bursty arrivals. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2610.31827 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-26T19:46:03.290Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).