Metraipolis › archive › dc_73fb6e18aa77

FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration — arXiv:2608.19659

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.19659
License (as archived)
CC0-1.0
Archived
2026-10-18T11:40:51.057Z
SHA-256 of the archived text
96bf3c9d3be3ac31ddea48a144c6d415d54cfe223b5fe61dfd7e752e272f1c0e
Stable URL of this page
https://metraipolis.com/documento/dc_73fb6e18aa77

How to cite

FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration — arXiv:2608.19659 — arXiv (en), https://arxiv.org/abs/2608.19659 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_73fb6e18aa77 (Metraipolis archive dc_73fb6e18aa77, sha256 96bf3c9d3be3ac31ddea48a144c6d415d54cfe223b5fe61dfd7e752e272f1c0e)
BibTeX
@misc{dc73fb6e18aa77,
  title = {FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration — arXiv:2608.19659},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.19659},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_73fb6e18aa77; sha256 96bf3c9d3be3ac31ddea48a144c6d415d54cfe223b5fe61dfd7e752e272f1c0e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19659. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load. Exhaustive profiling resolves this uncertainty, but measures many configurations that do not affect the final resource allocation. We present FleetSieve, which selects measurements according to their expected effect on a resource-coupled, SLO-aware fleet decision. FleetSieve models capacity and tail latency jointly, compares conservative and optimistic allocations, and stops when their remaining decision gap is below a specified tolerance. On a fixed H100 measurement grid for a 31B-parameter open-weight model, FleetSieve reaches the oracle aggregate decision using 22,200 GPU-seconds, 6.9% less than uniform random profiling in the fixed comparison. Across 200 random reveal orders, its mean saving over random profiling is 5.4% (95% bootstrap CI: 3.5-7.2%). The fixed-comparison saving is 21.5% for Chat, while FleetSieve does not use the fewest GPU-seconds for Code. Joint capacity and tail modeling also avoids selecting a configuration whose 46.4-second completion p99 violates a 30-second SLO. In a 16-GPU allocation, an incorrect sparse-profile decision loses up to 1.93 requests/s and 12.4 percentage points of max-min fulfillment. Boundary repeats and BurstGPT measurements support the observed load-dependent tail-latency mechanism.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-21 in cs.DC).
Categorie arXiv: Machine Learning (cs.LG) ; Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.19659 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T11:38:59.519Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).