Metraipolis › archive › dc_685bd44d4e39

FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction — arXiv:2608.14205

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.14205
License (as archived): CC0-1.0
Archived: 2026-10-19T02:20:38.266Z
SHA-256 of the archived text: 3d2020158fe7746b71f538e67e84d67bbb238fa3055f545fc27a5dd5ba84a0a8
Stable URL of this page: https://metraipolis.com/documento/dc_685bd44d4e39

How to cite

FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction — arXiv:2608.14205 — arXiv (en), https://arxiv.org/abs/2608.14205 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_685bd44d4e39 (Metraipolis archive dc_685bd44d4e39, sha256 3d2020158fe7746b71f538e67e84d67bbb238fa3055f545fc27a5dd5ba84a0a8)

BibTeX

@misc{dc685bd44d4e39,
  title = {FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction — arXiv:2608.14205},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14205},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_685bd44d4e39; sha256 3d2020158fe7746b71f538e67e84d67bbb238fa3055f545fc27a5dd5ba84a0a8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14205. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models. The most heavily loaded rank stalls global execution due to skewed routing distributions, directly increasing latency. While offline expert placement can alleviate persistent imbalance, practical multi-task serving workloads exhibit layer- and batch-dependent routing dynamics, making online load balancing indispensable. Existing approaches rely on routing statistics collected after each MoE router, requiring expert weight load or migration to begin only after routing decisions are available, consequently placing migration overhead on the inference critical path. In this work, we observe that online balancing can instead be largely overlapped with computation before target routing (e.g., attention), if routing distributions can be predicted accurately in advance. Therefore, we propose FreeBalance, a lossless online load-balancing framework that overlaps expert migration with preceding computation stages via residual workload prediction. FreeBalance leverages cross-layer similarities in hidden representations within the residual network to build a lightweight workload predictor. This enables proactive expert migration planning before routing decisions are available, creating substantial overlap between weight transfer and computation-heavy pre-routing stages. Furthermore, a cost model constrains the number of swaps to fully hide the synchronization overhead within the available window. Experiments across models and datasets show that FreeBalance reduces the max-to-mean rank load ratio by 32.8% and end-to-end prefill latency by 13.1%. Specifically, our method hides balancing overhead of an average of 5.1 experts per layer, which would otherwise account for about 8.5% of the critical-path latency.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.14205 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.791Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).