Metraipolis › archive › dc_bf6466ecd7a2

An Approximate Queueing Model of LLM Inference Serving for SLO-Driven Autoscaling — arXiv:2609.20957

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.20957
License (as archived): CC0-1.0
Archived: 2026-10-10T01:40:22.914Z
SHA-256 of the archived text: 0f25a74458393a7ccdd7cdf285f767ce30b3690d6c8a5197eb41b9d6d8583dc0
Stable URL of this page: https://metraipolis.com/documento/dc_bf6466ecd7a2

How to cite

An Approximate Queueing Model of LLM Inference Serving for SLO-Driven Autoscaling — arXiv:2609.20957 — arXiv (en), https://arxiv.org/abs/2609.20957 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf6466ecd7a2 (Metraipolis archive dc_bf6466ecd7a2, sha256 0f25a74458393a7ccdd7cdf285f767ce30b3690d6c8a5197eb41b9d6d8583dc0)

BibTeX

@misc{dcbf6466ecd7a2,
  title = {An Approximate Queueing Model of LLM Inference Serving for SLO-Driven Autoscaling — arXiv:2609.20957},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.20957},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf6466ecd7a2; sha256 0f25a74458393a7ccdd7cdf285f767ce30b3690d6c8a5197eb41b9d6d8583dc0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.20957. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Performance models of LLM servers support both latency evaluation and the design of controllers for autoscaling against service level objectives (SLOs) and for inference optimization. We model the multiplexed execution of prefill and decode operations with a tractable, approximate queueing model under Markovian assumptions. Three parameters characterize a model-accelerator pair, namely a baseline per-iteration overhead, a per-token compute cost, and a per-token key-value (KV) cache access cost. The model combines a mean-value analysis of per-iteration work with a state-dependent Markov chain for batch occupancy to predict mean time to first token (TTFT) and inter-token latency (ITL). We validate these predictions against measurements and show that the three parameters can be estimated from observed latencies. Over a grid of input and output lengths and arrival rates spanning light to moderate load, the relative error of the average ITL is about 5% for Llama-3.1-8B and 8% for Qwen2.5-14B running on an H100 GPU, and the corresponding TTFT errors are 14% and 16%. We then implement an autoscaling controller that uses the model to adjust inference-server replica counts as the workload changes. On an OpenShift cluster of H100 GPUs it tracks a fourfold load ramp under both latency targets, missing one in 7 of 127 control cycles, and its in-loop predictions carry median errors of at most 5% for TTFT and 9% for ITL. A decode-throughput analyzer from an existing autoscaler, which takes no latency target, misses 27 of 128 cycles under the same controller and load while provisioning 4% and 28% fewer replicas.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-21 in cs.PF).
Categorie arXiv: Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2609.20957 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:08:03.184Z dalla pagina https://arxiv.org/catchup/cs.PF/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).