Metraipolis › archive › dc_204d288f17e4

HW-Router: Hardware-Aware Routing for Scalable Multi-LLM Serving — arXiv:2608.14575

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.14575
License (as archived)
CC0-1.0
Archived
2026-10-18T19:55:31.290Z
SHA-256 of the archived text
e8ec4981d78b0877f1887c65be50477140dafd5073fa0ddbc95ce3f688f2588b
Stable URL of this page
https://metraipolis.com/documento/dc_204d288f17e4

How to cite

HW-Router: Hardware-Aware Routing for Scalable Multi-LLM Serving — arXiv:2608.14575 — arXiv (en), https://arxiv.org/abs/2608.14575 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_204d288f17e4 (Metraipolis archive dc_204d288f17e4, sha256 e8ec4981d78b0877f1887c65be50477140dafd5073fa0ddbc95ce3f688f2588b)
BibTeX
@misc{dc204d288f17e4,
  title = {HW-Router: Hardware-Aware Routing for Scalable Multi-LLM Serving — arXiv:2608.14575},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14575},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_204d288f17e4; sha256 e8ec4981d78b0877f1887c65be50477140dafd5073fa0ddbc95ce3f688f2588b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14575. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern large language model (LLM) serving platforms deploy multiple models across different GPUs, requiring routers to direct incoming queries to appropriate LLMs. However, existing routing approaches primarily rely on static model attributes such as size or FLOPs to estimate serving costs. This static cost modeling fails to capture the dynamic behavior of real deployments, where the same model can exhibit vastly different inference latencies depending on hardware type (e.g., H100 vs. V100), current system load (e.g., running and waiting queue lengths), and resource contention (e.g., KV-cache usage and GPU utilization). Such hardware-agnostic routing leads to suboptimal decisions, resulting in SLO violations, queue buildup, and underutilized GPUs. To address these challenges, we present HW-Router, a dynamic routing framework that integrates real-time hardware signals into model selection to enable accurate latency prediction and intelligent, SLO-aware routing decisions. Our approach incorporates model-specific features (architecture, size, input length) alongside hardware metrics including queue lengths, KV-cache utilization, and recent TTFT/TPOT performance, and uses a lightweight latency predictor to estimate per-model-per-GPU serving time. Evaluations across diverse workloads show that HW-Router achieves 3.4-3.9x lower end-to-end latency, 46-48 percentage points higher SLO attainment, 6-8x lower GPU load skew, and a 3.1-3.4x reduction in waiting-queue fraction compared to state-of-the-art router baselines, CARROT and IRT, with only ~200 us of additional routing overhead and no loss in output quality. These results highlight the importance of real-time hardware feedback for scalable, predictable, and well-balanced multi-LLM serving. Code is available at this https URL .

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-18 in cs.NI).
Categorie arXiv: Networking and Internet Architecture (cs.NI).
Pagina del lavoro: https://arxiv.org/abs/2608.14575 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T19:42:00.785Z dalla pagina https://arxiv.org/catchup/cs.NI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).