Metraipolis › archive › dc_cf000cffc3fc

Cross-Model Autoscaling for Shared LLM Serving — arXiv:2609.29160

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.29160
License (as archived)
CC0-1.0
Archived
2026-10-08T04:50:30.916Z
SHA-256 of the archived text
f2ce30c2406753b29811cbd772a00f4b6a1df6358d77f578599a90f5bdffab7c
Stable URL of this page
https://metraipolis.com/documento/dc_cf000cffc3fc

How to cite

Cross-Model Autoscaling for Shared LLM Serving — arXiv:2609.29160 — arXiv (en), https://arxiv.org/abs/2609.29160 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cf000cffc3fc (Metraipolis archive dc_cf000cffc3fc, sha256 f2ce30c2406753b29811cbd772a00f4b6a1df6358d77f578599a90f5bdffab7c)
BibTeX
@misc{dccf000cffc3fc,
  title = {Cross-Model Autoscaling for Shared LLM Serving — arXiv:2609.29160},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.29160},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cf000cffc3fc; sha256 f2ce30c2406753b29811cbd772a00f4b6a1df6358d77f578599a90f5bdffab7c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29160. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-model LLM serving is moving toward shared MaaS clusters, where co-hosted models compete for a fixed GPU budget while each model experiences time-varying demand and must satisfy its own latency SLO. Existing LLM autoscalers remain largely model-local: their signals expose local runtime activity or delayed latency outcomes, and their scale-up or provisioning decisions do not directly determine how shared capacity should be allocated across competing models. We present the Token-service-share Rebalancing Engine (TRE), a control-plane framework for hot-switched multi-model LLM serving. TRE introduces Token Service Share (TSS), a calibrated, demand-normalized signal that estimates effective token service per active or queued request and yields a comparable health score across heterogeneous models and SLO classes. Guided by TSS, TRE coordinates bounded receiver--donor capacity movement under a fixed GPU budget: it separates fast rescue from slower rebalancing and incrementally reallocates active replicas toward models with the largest calibrated service deficits. We implement TRE on a Kubernetes-based hot-switch serving stack without modifying the inference scheduler. Across seven LLM serving traces, TRE reduces P95 end-to-end latency by 11.9--79.0\% and P99 latency by 12.5--72.6\% compared with a state-of-the-art KV-cache-based reactive autoscaler running on the same hot-switch runtime. The gains hold on both targeted stress probes and production-derived conversation/code traces, where TRE reduces P95/P99 latency by 50.8/63.7\% and 79.0/72.6\%, respectively. These results show that effective hot-switched autoscaling requires not only fast replica actuation, but also calibrated service-deficit signals and coordinated cross-model capacity arbitration. Our code and artifacts are available at this https URL .

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-25 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2609.29160 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T04:28:24.116Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).