Metraipolis › archive › dc_32d4e242ed0c

Capacity Planning for Resource-Constrained Clinical LLM Serving — arXiv:2611.01152

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01152
License (as archived)
CC0-1.0
Archived
2026-11-03T10:22:10.637Z
SHA-256 of the archived text
a808e6e9dd0646fa8e95ec01c5439c064ef43872b5cca16364fa755e4afdea0b
Stable URL of this page
https://metraipolis.com/documento/dc_32d4e242ed0c

How to cite

Capacity Planning for Resource-Constrained Clinical LLM Serving — arXiv:2611.01152 — arXiv (en), https://arxiv.org/abs/2611.01152 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_32d4e242ed0c (Metraipolis archive dc_32d4e242ed0c, sha256 a808e6e9dd0646fa8e95ec01c5439c064ef43872b5cca16364fa755e4afdea0b)
BibTeX
@misc{dc32d4e242ed0c,
  title = {Capacity Planning for Resource-Constrained Clinical LLM Serving — arXiv:2611.01152},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01152},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_32d4e242ed0c; sha256 a808e6e9dd0646fa8e95ec01c5439c064ef43872b5cca16364fa755e4afdea0b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01152. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Deploying large language models (LLMs) under a fixed accelerator budget requires jointly deciding how computational capacity is allocated across model sizes and how incoming requests are routed across that capacity. Although prior work has extensively studied LLM routing and serving efficiency, the relative importance of fleet composition and routing under resource constraints remains less understood. We investigate this question using a controlled four-GPU serving environment with heterogeneous clinical NLP workloads spanning short factual verification, biomedical semantic matching, and long-context clinical summarization. We systematically vary fleet composition, routing policy, and offered load while holding the workload trace and hardware budget fixed. Across 24 configurations, descriptive main-effect shares attribute more variation to fleet composition than to the evaluated routing policies in time to first token, latency, and long-context coverage. All-small fleets provide the best responsiveness and throughput but substantially lower long-context coverage, whereas larger-model capacity improves coverage at the cost of serving efficiency. Heterogeneous fleets provide intermediate operating points but can introduce strong-model bottlenecks when large-model capacity is scarce. In a focused evaluation of this bottleneck, capacity-aware routing reduces mean latency and increases throughput while incurring a small reduction in long-context coverage. Within this measured regime, the results support prioritizing capacity allocation, with routing managing demand within the provisioned capacity.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2611.01152 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:19:49.810Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).