Metraipolis › archive › dc_d0259aff1010

MegaFlux: Skew-Resilient MoE Megakernels via Pipelined Expert Replication — arXiv:2610.00671

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00671
License (as archived)
CC0-1.0
Archived
2026-10-06T14:02:18.586Z
SHA-256 of the archived text
8b9badd2473ef27c1b1f72cb8f0a5b15f6e8bbab3f13745b92faea7fe2cb3c84
Stable URL of this page
https://metraipolis.com/documento/dc_d0259aff1010

How to cite

MegaFlux: Skew-Resilient MoE Megakernels via Pipelined Expert Replication — arXiv:2610.00671 — arXiv (en), https://arxiv.org/abs/2610.00671 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d0259aff1010 (Metraipolis archive dc_d0259aff1010, sha256 8b9badd2473ef27c1b1f72cb8f0a5b15f6e8bbab3f13745b92faea7fe2cb3c84)
BibTeX
@misc{dcd0259aff1010,
  title = {MegaFlux: Skew-Resilient MoE Megakernels via Pipelined Expert Replication — arXiv:2610.00671},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00671},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d0259aff1010; sha256 8b9badd2473ef27c1b1f72cb8f0a5b15f6e8bbab3f13745b92faea7fe2cb3c84},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00671. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-experts (MoE) megakernels fuse expert-parallel communication with expert computation. However, under fixed expert placement, routing skew creates GPU stragglers: overloaded GPUs determine layer latency while others sit idle. Replicating hot experts can shift work to underloaded GPUs, but dynamic replicas introduce additional work: replicas must receive expert weights to execute and, during training, their partial weight gradients must be reduced at the expert owners. We present MegaFlux, which makes expert replication a runtime decision and pipelines the communication induced by replication within persistent MoE execution. An on-device planner jointly selects replica locations and assigns tile-aligned token blocks under a per-GPU replica budget, leaving router outputs unchanged. The forward and backward megakernels realize pipelined expert replication: replicas begin computation as their required weights arrive, while backward overlaps replica-gradient reduction with ongoing expert computation. MegaFlux extends TensorRT-LLM's CuTeDSL MegaMoE forward kernel and introduces a new backward MoE megakernel. Across 147 configurations per direction on eight NVIDIA B200 GPUs, MegaFlux achieves geometric-mean speedups of $1.45\times$ for forward and $1.28\times$ for backward over the same megakernels with fixed placement, peaking at $2.14\times$ and $2.64\times$. In ablations, pipelining hides $56$--$76$% of replica-weight transfer cost in forward and $91$--$100$% of combined weight-transfer and replica-gradient-reduction cost in backward, yielding up to $13.2$% and $26.7$% additional layer-latency reductions over the same replication plans with these operations executed separately. Integrated into vLLM for DeepSeek-V4-Pro prefill, MegaFlux delivers $1.13$--$1.26\times$ median end-to-end speedups over fixed placement.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-10-02 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2610.00671 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:22:22.536Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).