Metraipolis › archive › dc_eb331b47c3b7

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving — arXiv:2609.18112

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18112
License (as archived)
CC0-1.0
Archived
2026-10-15T13:53:29.687Z
SHA-256 of the archived text
4e978b7fd7661f1f640f753057a0bef154bf01836c9cdfa728f85dd21697a438
Stable URL of this page
https://metraipolis.com/documento/dc_eb331b47c3b7

How to cite

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving — arXiv:2609.18112 — arXiv (en), https://arxiv.org/abs/2609.18112 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb331b47c3b7 (Metraipolis archive dc_eb331b47c3b7, sha256 4e978b7fd7661f1f640f753057a0bef154bf01836c9cdfa728f85dd21697a438)
BibTeX
@misc{dceb331b47c3b7,
  title = {Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving — arXiv:2609.18112},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18112},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb331b47c3b7; sha256 4e978b7fd7661f1f640f753057a0bef154bf01836c9cdfa728f85dd21697a438},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18112. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel {\delta}-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + {\delta} time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-17 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.18112 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:48:52.617Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).