Metraipolis › archive › dc_e334c0d909cc

Rethinking KV Cache Eviction for Agentic LLM Serving — arXiv:2611.00391

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00391
License (as archived)
CC0-1.0
Archived
2026-11-03T18:26:34.024Z
SHA-256 of the archived text
e6e197e97753a616ed73b8bfc6fd71da76a37175c37ea256f80b0a592c015e88
Stable URL of this page
https://metraipolis.com/documento/dc_e334c0d909cc

How to cite

Rethinking KV Cache Eviction for Agentic LLM Serving — arXiv:2611.00391 — arXiv (en), https://arxiv.org/abs/2611.00391 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e334c0d909cc (Metraipolis archive dc_e334c0d909cc, sha256 e6e197e97753a616ed73b8bfc6fd71da76a37175c37ea256f80b0a592c015e88)
BibTeX
@misc{dce334c0d909cc,
  title = {Rethinking KV Cache Eviction for Agentic LLM Serving — arXiv:2611.00391},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00391},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e334c0d909cc; sha256 e6e197e97753a616ed73b8bfc6fd71da76a37175c37ea256f80b0a592c015e88},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00391. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Key-value (KV) caches impose substantial memory pressure on large language model serving systems, requiring cache eviction when capacity is insufficient. Existing mechanisms typically evict KV from the ends of cached sequences to preserve prefix sharing across requests. However, agent workloads repeatedly reuse long histories across successive turns. Recomputing the KV cache for tokens near the end of a sequence is substantially more expensive than for earlier tokens because their attention computation involves longer histories. To address this problem, we propose preserving shared prefixes and evicting the KV cache from the beginning of the remaining history while retaining the cached suffix. We build Penelope around this strategy by co-designing the eviction policy, KV recomputation, and unified GPU-CPU cache management. Returning requests reuse the KV cache retained in GPU or CPU memory and recompute only missing segments while preserving reuse of the cached suffix. We implement Penelope in SGLang and evaluate it on three Qwen3 model sizes using real agent traces. Compared with SGLang, Penelope reduces mean time to first token by up to 60% and job completion time by up to 37% with a GPU-only cache, and by up to 76% and 60% with a two-tier GPU-CPU cache.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.00391 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:22.791Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).