Metraipolis › archive › dc_728ecbc14720

ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration — arXiv:2611.04032

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04032
License (as archived)
CC0-1.0
Archived
2026-11-04T18:33:47.670Z
SHA-256 of the archived text
6416374867bf08826eae05eb51d9e0dfa6d24d11e8d9d127c79512940e9c2328
Stable URL of this page
https://metraipolis.com/documento/dc_728ecbc14720

How to cite

ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration — arXiv:2611.04032 — arXiv (en), https://arxiv.org/abs/2611.04032 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_728ecbc14720 (Metraipolis archive dc_728ecbc14720, sha256 6416374867bf08826eae05eb51d9e0dfa6d24d11e8d9d127c79512940e9c2328)
BibTeX
@misc{dc728ecbc14720,
  title = {ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration — arXiv:2611.04032},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04032},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_728ecbc14720; sha256 6416374867bf08826eae05eb51d9e0dfa6d24d11e8d9d127c79512940e9c2328},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04032. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Serving an LLM efficiently depends on the engine, how the model is split across GPUs, precision, scheduler settings, speculative decoding, KV-cache management, and prefill-decode disaggregation. The best configuration depends on the model, hardware, workload, and objective. Operators usually find it by trial and error. Exhaustively measuring the candidates in our study would take about 25,000 GPU-hours. ROCm InferaSim predicts serving performance from a few short measurements. Choices such as batch size, prompt length, and GPU count change the amount of work per GPU without changing the kernels. ROCm InferaSim benchmarks each distinct set of kernels once, on at most one node. We call this measurement an anchor. Analytical laws carry that anchor to other deployments that run the same kernels. For multi-node deployments, ROCm InferaSim reduces the configuration to a single-node equivalent, measures it, and scales the result analytically. A simulator replays multi-turn traffic, and a tuning agent searches configurations using the predictions. On AMD Instinct MI355X GPUs with vLLM, SGLang, and ATOM, 11 anchors took 9.1 GPU-hours, against 204 GPU-hours to measure the 226 fixed-length configurations they cover. On 172 configurations, time per output token has 8.1% median error. On a multi-turn coding workload, throughput is within 8%. For 159 disaggregated deployments, none used as an anchor, throughput is within 5.5%. A five-candidate short list contains a deployment within 12% of the best feasible pairing of model and latency target.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.04032 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:37.261Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).