Metraipolis › archive › dc_4e520c4d9b10

Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling — arXiv:2610.09372

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.09372
License (as archived)
CC0-1.0
Archived
2026-10-08T11:26:03.225Z
SHA-256 of the archived text
4432a6ee5c75339829359254929ab9a9930bb99e9efe815e82cddb8b2f54aebd
Stable URL of this page
https://metraipolis.com/documento/dc_4e520c4d9b10

How to cite

Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling — arXiv:2610.09372 — arXiv (en), https://arxiv.org/abs/2610.09372 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e520c4d9b10 (Metraipolis archive dc_4e520c4d9b10, sha256 4432a6ee5c75339829359254929ab9a9930bb99e9efe815e82cddb8b2f54aebd)
BibTeX
@misc{dc4e520c4d9b10,
  title = {Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling — arXiv:2610.09372},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09372},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e520c4d9b10; sha256 4432a6ee5c75339829359254929ab9a9930bb99e9efe815e82cddb8b2f54aebd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09372. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) layers replace the feed-forward block of a Transformer with E expert networks, and each token is routed to k of these experts. Under expert parallelism (EP) the experts are distributed across GPUs, and every MoE layer runs all-to-all collectives in the forward and backward passes to dispatch tokens to their experts and then combine the results. On a cluster with 8 AMD Instinct MI300X GPUs per node, these collectives can take 45% of the training step at EP32 with top-2 routing and 60% with top-6 routing. We find that early in pretraining routers have already learned to assign tokens to experts in correlated patterns, both within a layer and across layers. At top-2, 0.8% of the expert pairs in a layer are selected together by 42% of tokens, and the experts a token selects at one layer predict the experts it selects at the next layer. We use these correlations to keep more token--expert assignments on the token's own GPU, which reduces communication across GPUs and across nodes. Correlated expert placement puts experts that are often selected together on the same GPU. Combined with a dispatcher that sends each token to each GPU once, it removes up to 58% of dispatched rows. Token shuffling applies when sequence parallelism shards tokens across the EP group. It moves each token to the GPU predicted to hold its next-layer experts during the reduce-scatter that follows attention. On one node this raises the share of token--expert assignments served on the token's GPU from 12.5% to 59%. In Megatron-LM, across EP degrees from 8 to 64 with top-2 and top-6 routing, the two methods reduce all-to-all time by 1.16-2.63X and end-to-end step time by up to 1.41X. Neither method changes the models' underlying routing decisions or expert parameters.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2610.09372 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:24:06.314Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).