Metraipolis › archive › dc_92b01171fdfc
Range: Decoupling KV-Cache Reuse Granularity from Storage Work — arXiv:2610.21241
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.21241
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T10:44:27.655Z
- SHA-256 of the archived text
cddac02fc2107c117151f911b957eeffa914704142dc01e45b6247d747102208- Stable URL of this page
https://metraipolis.com/documento/dc_92b01171fdfc
How to cite
Range: Decoupling KV-Cache Reuse Granularity from Storage Work — arXiv:2610.21241 — arXiv (en), https://arxiv.org/abs/2610.21241 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_92b01171fdfc (Metraipolis archive dc_92b01171fdfc, sha256 cddac02fc2107c117151f911b957eeffa914704142dc01e45b6247d747102208)
BibTeX
@misc{dc92b01171fdfc,
title = {Range: Decoupling KV-Cache Reuse Granularity from Storage Work — arXiv:2610.21241},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.21241},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_92b01171fdfc; sha256 cddac02fc2107c117151f911b957eeffa914704142dc01e45b6247d747102208},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21241. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM serving reuses cached key-value (KV) state to skip prefill, and growing contexts push that state past GPU HBM and CPU DRAM onto NVMe SSDs, placing prefix restore on the critical path of time-to-first-token (TTFT). We find that on a well-configured host-issued path the SSD is not the bottleneck: restoring the same 6 GiB of KV takes 2.1 s as 192 objects but 3.0 s as 6,144 objects, and more under layerwise loading, because every layer of the storage stack - lookup, allocation, I/O submission, GPU delivery, store, index - charges per logical object while the bytes stay the same. This cost grows with the KV footprint: longer contexts, deeper models under layerwise loading, and tensor parallelism each multiply the object count. Prior systems move this per-object work from the CPU to the GPU; the work remains per object. This paper presents Range, a KV-cache storage engine for vLLM and LMCache built on one principle: every cost on the storage path scales with bytes or physical batches, never with logical objects. Range tickets replace per-key lookup and ownership with token metadata and physical leases; byte stripes make the on-media layout independent of reuse granularity; completion-owned GPU transfer banks batch delivery and extraction and release each layer to attention as it lands. Range keeps the kernel cuFile path and changes four lines of vLLM. On a fixed 6 GiB restore, Range's median TTFT stays at 1.79-1.80 s across a 32x range of object counts and under layerwise delivery, within 70 ms of the prefix already being resident in GPU memory, while tuned in-process LMCache rises from 2.12 s to 3.01 s. Range improves on the fastest multiprocess DRAM-bounced path by 0.44 s at coarse granularity while moving no payload through host memory, and holds layerwise reuse at the cost of coarse-grained reuse, where the best stock path takes 2.24 s. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.OS, nuovi lavori del giorno). Categorie arXiv: Operating Systems (cs.OS). Pagina del lavoro: https://arxiv.org/abs/2610.21241 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:55:22.196Z dalla pagina https://arxiv.org/list/cs.OS/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).