Metraipolis › archive › dc_9fa2b4dfb751

The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems — arXiv:2609.27746

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.27746
License (as archived): CC0-1.0
Archived: 2026-10-08T10:06:24.622Z
SHA-256 of the archived text: 78cbda4f2c5d31f6919226953779ddb031b416e360a0ba0a372f3137d741e85a
Stable URL of this page: https://metraipolis.com/documento/dc_9fa2b4dfb751

How to cite

The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems — arXiv:2609.27746 — arXiv (en), https://arxiv.org/abs/2609.27746 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9fa2b4dfb751 (Metraipolis archive dc_9fa2b4dfb751, sha256 78cbda4f2c5d31f6919226953779ddb031b416e360a0ba0a372f3137d741e85a)

BibTeX

@misc{dc9fa2b4dfb751,
  title = {The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems — arXiv:2609.27746},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27746},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9fa2b4dfb751; sha256 78cbda4f2c5d31f6919226953779ddb031b416e360a0ba0a372f3137d741e85a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27746. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Prefix caching is critical for efficient large language model (LLM) serving, particularly for agentic workloads that repeatedly invoke the model with a growing conversation and tool-use history. By reusing the key-value (KV) states of previously processed prefixes, prefix caching avoids redundant prefill computation. Its effectiveness, however, depends on retaining a sufficiently large set of KV cache states. Provisioning enough cache to preserve all historical KV states is prohibitively expensive and often unnecessary, whereas insufficient capacity can substantially degrade the cache hit rate. Determining the KV cache working set, defined as the minimum cache capacity required to achieve a target hit rate, is therefore essential for efficient cache provisioning and system design. We present KVSET, an online analyzer that estimates the KV cache working set of LLM serving workloads. KVSET uses the Mattson stack algorithm to efficiently estimate cache hit rates across a wide range of cache capacities. For each KV cache page, KVSET computes its LRU stack distance and compares it with the page number of each candidate capacity. This comparison determines whether the page would be a hit at each capacity without independently simulating every capacity configuration. KVSET therefore substantially reduces the computational and memory overhead of conventional capacity-by-capacity simulation and makes online working-set analysis practical. KVSET further determines the minimum cache capacity based on the maximum LRU depth among the prefix pages required to achieve the target hit rate. We validate KVSET using traces collected from production LLM workloads and show that its estimates closely match measurements from real cache deployments. The open-source implementation supports both online request processing and offline trace replay.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-24 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2609.27746 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T09:13:37.342Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).