Metraipolis › archive › dc_cf000cffc3fc
Cross-Model Autoscaling for Shared LLM Serving — arXiv:2609.29160
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.29160
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T04:50:30.916Z
- SHA-256 of the archived text
f2ce30c2406753b29811cbd772a00f4b6a1df6358d77f578599a90f5bdffab7c- Stable URL of this page
https://metraipolis.com/documento/dc_cf000cffc3fc
How to cite
Cross-Model Autoscaling for Shared LLM Serving — arXiv:2609.29160 — arXiv (en), https://arxiv.org/abs/2609.29160 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cf000cffc3fc (Metraipolis archive dc_cf000cffc3fc, sha256 f2ce30c2406753b29811cbd772a00f4b6a1df6358d77f578599a90f5bdffab7c)
BibTeX
@misc{dccf000cffc3fc,
title = {Cross-Model Autoscaling for Shared LLM Serving — arXiv:2609.29160},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.29160},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cf000cffc3fc; sha256 f2ce30c2406753b29811cbd772a00f4b6a1df6358d77f578599a90f5bdffab7c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29160. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multi-model LLM serving is moving toward shared MaaS clusters, where co-hosted models compete for a fixed GPU budget while each model experiences time-varying demand and must satisfy its own latency SLO. Existing LLM autoscalers remain largely model-local: their signals expose local runtime activity or delayed latency outcomes, and their scale-up or provisioning decisions do not directly determine how shared capacity should be allocated across competing models. We present the Token-service-share Rebalancing Engine (TRE), a control-plane framework for hot-switched multi-model LLM serving. TRE introduces Token Service Share (TSS), a calibrated, demand-normalized signal that estimates effective token service per active or queued request and yields a comparable health score across heterogeneous models and SLO classes. Guided by TSS, TRE coordinates bounded receiver--donor capacity movement under a fixed GPU budget: it separates fast rescue from slower rebalancing and incrementally reallocates active replicas toward models with the largest calibrated service deficits. We implement TRE on a Kubernetes-based hot-switch serving stack without modifying the inference scheduler. Across seven LLM serving traces, TRE reduces P95 end-to-end latency by 11.9--79.0\% and P99 latency by 12.5--72.6\% compared with a state-of-the-art KV-cache-based reactive autoscaler running on the same hot-switch runtime. The gains hold on both targeted stress probes and production-derived conversation/code traces, where TRE reduces P95/P99 latency by 50.8/63.7\% and 79.0/72.6\%, respectively. These results show that effective hot-switched autoscaling requires not only fast replica actuation, but also calibrated service-deficit signals and coordinated cross-model capacity arbitration. Our code and artifacts are available at this https URL . Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-25 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2609.29160 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T04:28:24.116Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).