Metraipolis › archive › dc_2b3b06017f89

TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes — arXiv:2608.13057

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13057
License (as archived)
CC0-1.0
Archived
2026-10-19T04:18:19.736Z
SHA-256 of the archived text
4d9f1e61ac27277799b7e4d4bbe4ebe583604606bb6d23a8cd2164546908ab4a
Stable URL of this page
https://metraipolis.com/documento/dc_2b3b06017f89

How to cite

TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes — arXiv:2608.13057 — arXiv (en), https://arxiv.org/abs/2608.13057 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2b3b06017f89 (Metraipolis archive dc_2b3b06017f89, sha256 4d9f1e61ac27277799b7e4d4bbe4ebe583604606bb6d23a8cd2164546908ab4a)
BibTeX
@misc{dc2b3b06017f89,
  title = {TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes — arXiv:2608.13057},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13057},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2b3b06017f89; sha256 4d9f1e61ac27277799b7e4d4bbe4ebe583604606bb6d23a8cd2164546908ab4a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13057. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU. Dispatchers balance token counts (EPLB, LPLB, UltraEP) or activated-expert counts (METRO), assuming expert time is linear in one. Measurements on two datacenter GPU generations show it is neither: below $n^* \approx 156$--$168$ tokens, HBM weight streaming dominates---cost attaches to $activated replicas$, not tokens; above it, grouped GEMM rounds tokens to 128-tile $M$-tiles, so $splitting$ an expert adds padded compute. A max-affine profile $t=\max(a+bG,\,c+\beta N)$ captures both regimes. Realistic decode batches hold hot experts in the linear regime and cold in the flat $simultaneously$; recorded batches show proxy dispatches differ by $1.4$--$1.6\times$ in modeled block time (p95 up to $1.7\times$), and $which$ proxy wins flips with the regime. We formalize per-batch dispatch as a fixed-charge makespan problem---NP-hard on two fully replicated GPUs, polynomial in degenerate limits---and present TEMPO, a makespan-aware dispatcher solving it in milliseconds off the critical path; its SGLang integration runs out-of-process and fuses dispatch with count collection into one in-graph kernel. Anchored by an 8-GPU Testbed A microbenchmark, TEMPO stays within $1\%$ of the best fixed baseline everywhere and wins by up to $15.5\%$ where regimes mix. End-to-end on Testbed B, Qwen3-235B (inside the win region) gains $4$--$6\%$ throughput and cuts p99 latency by $\sim 15.6\%$; DeepSeek-V3 (outside, communication-dominated) shows only mechanism cost. A phase diagram, not a universal win, is the claim: it predicts both outcomes before deployment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.AI).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Science and Game Theory (cs.GT).
Pagina del lavoro: https://arxiv.org/abs/2608.13057 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:13:06.733Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).