Metraipolis › archive › dc_63b80762dbf7

Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving — arXiv:2608.16336

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.16336
License (as archived)
CC0-1.0
Archived
2026-10-18T19:02:12.562Z
SHA-256 of the archived text
93976aee6764304d4d119278b33668a77dfb4cc2495fdf49613013362e46a13d
Stable URL of this page
https://metraipolis.com/documento/dc_63b80762dbf7

How to cite

Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving — arXiv:2608.16336 — arXiv (en), https://arxiv.org/abs/2608.16336 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_63b80762dbf7 (Metraipolis archive dc_63b80762dbf7, sha256 93976aee6764304d4d119278b33668a77dfb4cc2495fdf49613013362e46a13d)
BibTeX
@misc{dc63b80762dbf7,
  title = {Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving — arXiv:2608.16336},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.16336},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_63b80762dbf7; sha256 93976aee6764304d4d119278b33668a77dfb4cc2495fdf49613013362e46a13d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.16336. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing. Llumnix introduced a dynamic, migration-capable multi-instance scheduler for LLM inference that achieves load balancing, defragmentation, prioritization, and auto-scaling through a unified "freeness" metric. However, Llumnix's priority model is restricted to two levels (high and normal), an abstraction too coarse to express the richer SLA classes common in production deployments. In this work, we extend Llumnix's priority model to support an arbitrary number of tiers and evaluate the effects of this extension under three realistic priority distributions (uniform, Gaussian, enterprise) using Vidur, a high-fidelity LLM inference simulator. We implement per-tier headroom with exponential decay, tier-aware dispatch ordering, and the full Llumnix migration pipeline inside Vidur's hierarchical scheduling framework. We compare our extended scheduler against INFaaS (global routing baseline), vLLM, Orca, and Sarathi-Serve (per-replica baselines), sweeping priority levels from 1 to 10. Our experiments demonstrate that four priority tiers yields the best cost-effectiveness tradeoff, achieving prefill mean speedups of up to 8.3x and end-to-end P99 speedups of up to 3.1x over INFaaS with cost-per-latency improvements of 46 to 68%, while preserving strong SLO differentiation across tiers. We further show that the system sustains these gains at 10 priority levels without tail latency collapse, with overhead concentrated in the prefill phase.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-18 in cs.DC).
Categorie arXiv: Hardware Architecture (cs.AR) ; Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.16336 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T18:07:16.091Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).