Metraipolis › archive › dc_315480cb3d86

Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts — arXiv:2609.28053

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.28053
License (as archived)
CC0-1.0
Archived
2026-10-08T10:05:42.759Z
SHA-256 of the archived text
84842c066fa6ba9b844a42801a0b19e66cd296193bc7561b848ac5d0630c0b2f
Stable URL of this page
https://metraipolis.com/documento/dc_315480cb3d86

How to cite

Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts — arXiv:2609.28053 — arXiv (en), https://arxiv.org/abs/2609.28053 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_315480cb3d86 (Metraipolis archive dc_315480cb3d86, sha256 84842c066fa6ba9b844a42801a0b19e66cd296193bc7561b848ac5d0630c0b2f)
BibTeX
@misc{dc315480cb3d86,
  title = {Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts — arXiv:2609.28053},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.28053},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_315480cb3d86; sha256 84842c066fa6ba9b844a42801a0b19e66cd296193bc7561b848ac5d0630c0b2f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.28053. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) training requires global load balance to prevent expert under-utilization and local balance for efficient expert-parallel execution. Existing distributed Quantile Balancing (QB) uses shard-dependent or approximate global quantiles, while token-independent expert biases cannot ensure microbatch-level balance. We introduce Exact Quantile Balancing (EQB), which computes exact global-batch BF16 quantiles with negligible communication, and Load-Error Injection (LEI), which injects local load errors directly into router-score gradients. On 7.5B-parameter MoEs trained for up to 500B tokens, EQB improves global balance and downstream performance over naive QB, while LEI improves local balance and outperforms the GShard loss at comparable quality.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.28053 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T09:11:49.694Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).