Metraipolis › archive › dc_d8367e1c8634

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — arXiv:2608.06557

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06557
License (as archived)
CC0-1.0
Archived
2026-10-19T09:47:46.561Z
SHA-256 of the archived text
d557d1a2a11069aed9c5a46656717e9c08ef28081d68105d0cc59584704f4c95
Stable URL of this page
https://metraipolis.com/documento/dc_d8367e1c8634

How to cite

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — arXiv:2608.06557 — arXiv (en), https://arxiv.org/abs/2608.06557 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d8367e1c8634 (Metraipolis archive dc_d8367e1c8634, sha256 d557d1a2a11069aed9c5a46656717e9c08ef28081d68105d0cc59584704f4c95)
BibTeX
@misc{dcd8367e1c8634,
  title = {Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — arXiv:2608.06557},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06557},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d8367e1c8634; sha256 d557d1a2a11069aed9c5a46656717e9c08ef28081d68105d0cc59584704f4c95},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06557. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests. LLM serving platforms today define response-latency service-level objectives, even though requests within the same service can differ by orders of magnitude in input length, generation length, execution cost, and the availability of reusable KV-cache state. As a result, requests governed by the same service level objective have different urgency: after accounting for the time required to execute them, some have substantial latency headroom while others have almost none. We define this headroom---the difference between a request's service level objective and its predicted remaining service time---as its per-request latency budget. We present Cascade, an LLM serving system that estimates and continuously updates this budget from request characteristics, KV-cache state, and current system load. Unlike prior SLO-aware schedulers that use deadlines to govern request ordering alone, Cascade uses a single per-request budget to jointly coordinate request scheduling and KV-cache management across the memory hierarchy. Its scheduler prioritizes requests with little remaining budget, while its memory manager uses the same budget to decide whether non-resident KV state should be restored or prefetched from a deeper tier, retained in HBM, or recomputed. By directing queueing and data-movement overhead toward requests that can absorb it, Cascade improves SLO-satisfied goodput while preserving fairness across heterogeneous request classes. On production traces across three large language models, Cascade improves goodput by up to2.4x and reduces SLO violations by 40% relative to the default vLLM first-come, first-served scheduler.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-10 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.06557 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:47:05.158Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).