Metraipolis › archive › dc_9ccfac6526ec

HiState: Hierarchical Prefix Cache Management for Hybrid LLM Serving in Agentic Workloads — arXiv:2610.41508

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41508
License (as archived): CC0-1.0
Archived: 2026-11-02T07:44:55.201Z
SHA-256 of the archived text: b4b9cc1e95308b85521ee9a4c1db496b7269b5feddab3570c1b231e9f08d498d
Stable URL of this page: https://metraipolis.com/documento/dc_9ccfac6526ec

How to cite

HiState: Hierarchical Prefix Cache Management for Hybrid LLM Serving in Agentic Workloads — arXiv:2610.41508 — arXiv (en), https://arxiv.org/abs/2610.41508 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9ccfac6526ec (Metraipolis archive dc_9ccfac6526ec, sha256 b4b9cc1e95308b85521ee9a4c1db496b7269b5feddab3570c1b231e9f08d498d)

BibTeX

@misc{dc9ccfac6526ec,
  title = {HiState: Hierarchical Prefix Cache Management for Hybrid LLM Serving in Agentic Workloads — arXiv:2610.41508},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41508},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9ccfac6526ec; sha256 b4b9cc1e95308b85521ee9a4c1db496b7269b5feddab3570c1b231e9f08d498d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41508. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hybrid large language models combine full-attention and recurrent linear layers to reduce inference costs. Prefix caching can further reduce redundant computation across requests, but a recurrent checkpoint cannot recover an earlier state corresponding to a shorter prefix. Existing hybrid prefix caching systems retain checkpoints at selected boundaries to support common reuse patterns in conventional conversations. However, agent workloads modify and revisit earlier contexts, exposing reusable prefixes whose checkpoints were never saved or have already been evicted. A straightforward approach is to retain more checkpoints to reduce recomputation, but their cumulative footprint can displace reusable prefixes from other sessions and cause additional cache misses. To address these problems, we present HiState, a prefix cache management system for hybrid models serving agent workloads. Our key observation is that prefix reuse accesses all matching key-value (KV) entries but only one recurrent checkpoint, regardless of how many checkpoints are retained. This asymmetry allows HiState to retain historical checkpoints on SSD while prioritizing fast memory for KV entries: each hit retrieves only one checkpoint whose size is independent of context length. Within this hierarchy, we propose adaptive interval control to balance checkpoint storage against recomputation, and unified eviction to coordinate checkpoint sparsification with reclamation of stale trajectories. We implement HiState on top of vLLM and evaluate it on two public coding-agent trace datasets and controlled microbenchmarks. On real agent traces, HiState achieves up to 5.73x and 5.44x speedups in p99 TTFT over vLLM-Marconi and SGLang-HiCache, respectively.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2610.41508 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:37:30.163Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).