Metraipolis › archive › dc_a8014ee77bcb

Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving — arXiv:2609.16206

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.16206
License (as archived)
CC0-1.0
Archived
2026-10-15T18:31:59.231Z
SHA-256 of the archived text
67e1092a9bb29a58f6f0a1e3c407a72d70547a2069ce6d6de6d38246f8d744dd
Stable URL of this page
https://metraipolis.com/documento/dc_a8014ee77bcb

How to cite

Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving — arXiv:2609.16206 — arXiv (en), https://arxiv.org/abs/2609.16206 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a8014ee77bcb (Metraipolis archive dc_a8014ee77bcb, sha256 67e1092a9bb29a58f6f0a1e3c407a72d70547a2069ce6d6de6d38246f8d744dd)
BibTeX
@misc{dca8014ee77bcb,
  title = {Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving — arXiv:2609.16206},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.16206},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a8014ee77bcb; sha256 67e1092a9bb29a58f6f0a1e3c407a72d70547a2069ce6d6de6d38246f8d744dd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16206. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle each request. We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class. We develop the policy in a discrete event simulator and validate it on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. All workloads run at measured saturation. Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic. It also shows the lowest variance across traces. It beats round robin and the length heuristic on all three traces and least loaded on two. On the third, it trails by 0.003, within run to run noise. Hardware calibration matters: simulator derived constants cost 4.5 goodput points and roughly 40 percent of the tail latency advantage, reducing the scorer to little more than queue counting. Benefits grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough. Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better. With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.16206 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:03:09.413Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).