Metraipolis › archive › dc_3df35bbc8cfc

When Cache Pressure Becomes a Trap: Metastability in Agent Serving — arXiv:2611.02543

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02543
License (as archived): CC0-1.0
Archived: 2026-11-03T11:12:42.573Z
SHA-256 of the archived text: dd379c50b859e8369d263580f2f801be86963affa035fbb621731dd7e1d7d0c7
Stable URL of this page: https://metraipolis.com/documento/dc_3df35bbc8cfc

How to cite

When Cache Pressure Becomes a Trap: Metastability in Agent Serving — arXiv:2611.02543 — arXiv (en), https://arxiv.org/abs/2611.02543 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3df35bbc8cfc (Metraipolis archive dc_3df35bbc8cfc, sha256 dd379c50b859e8369d263580f2f801be86963affa035fbb621731dd7e1d7d0c7)

BibTeX

@misc{dc3df35bbc8cfc,
  title = {When Cache Pressure Becomes a Trap: Metastability in Agent Serving — arXiv:2611.02543},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02543},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3df35bbc8cfc; sha256 dd379c50b859e8369d263580f2f801be86963affa035fbb621731dd7e1d7d0c7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02543. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM agents repeatedly send a growing conversation to a model, making prefix reuse central to efficient serving. We show that a brief surge can disrupt this reuse and trigger metastable failure in prefill--decode (PD) disaggregated engines: service remains degraded after arrivals return to a load the engine previously sustained. On SGLang, a ten-minute surge reduced goodput by about 88% over the following hour. Removing the residual burst sessions restored service within minutes. The sustaining mechanism is population-cache feedback. Slower calls prolong sessions and enlarge their aggregate context, or working set; the resulting cache misses increase recomputation and further delay departures. A fluid model identifies the resource conditions for this feedback and explains how cache capacity, tool time and base load affect vulnerability. Across 36 simulated deployments, it flags all 17 cache-first cases. Controlled simulations hold the post-surge state fixed and show that changing subsequent arrivals alone reverses the recovery outcome. Once the cache collapses, low-reuse departure capacity sets the scale of the load reduction needed for recovery. Guided by these dynamics, we build Basin, which gives each session a fixed priority, reserves their predicted peak contexts against an adaptive admission budget, extends the cache with idle decode memory, and sheds excess demand. On real H200 1P3D deployments under surges, including long-context SWE-bench Pro sessions, an arrival-time prototype achieves 1.20--1.70x the goodput of Continuum and 2.35--3.59x that of HiCache alone. In simulation, the predictor-driven policy delivers 1.20--2.05x the strongest baseline's goodput across six surge scenarios.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.02543 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:00:07.160Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).