Metraipolis › archive › dc_48cebbdf14d2

Muon Makes the First Mixture-of-Experts Layer Harder to Balance — arXiv:2610.37914

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.37914
License (as archived): CC0-1.0
Archived: 2026-10-31T06:17:14.790Z
SHA-256 of the archived text: b626176527593ac24323ceed119266e35f46006b98b43d0227ffea73f12764e7
Stable URL of this page: https://metraipolis.com/documento/dc_48cebbdf14d2

How to cite

Muon Makes the First Mixture-of-Experts Layer Harder to Balance — arXiv:2610.37914 — arXiv (en), https://arxiv.org/abs/2610.37914 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_48cebbdf14d2 (Metraipolis archive dc_48cebbdf14d2, sha256 b626176527593ac24323ceed119266e35f46006b98b43d0227ffea73f12764e7)

BibTeX

@misc{dc48cebbdf14d2,
  title = {Muon Makes the First Mixture-of-Experts Layer Harder to Balance — arXiv:2610.37914},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.37914},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_48cebbdf14d2; sha256 b626176527593ac24323ceed119266e35f46006b98b43d0227ffea73f12764e7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.37914. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router. In models with 55M and 108M active parameters, we measure how the optimizer changes the span of these biases, the largest minus the smallest in a layer. The span grows when all tokens favor the same experts, so it shows how hard a layer is to balance. At learning rates tuned on shorter runs, Muon raised the span of the first MoE layer at every seed, roughly doubling it on average, while the other layers changed much less. The rise came from Muon on the rest of the network, whichever optimizer trained the router. In those runs, the inputs to the first router shared a direction, and soon after the warmup its busiest expert averaged four to seven times its fair share. Over the first 3,000 steps, subtracting the batch mean from these inputs removed this overload while barely changing the loss. On the smaller model, halving Muon's learning rate removed most of the rise and slightly improved the loss. Moving only the router to AdamW made the loads shift more during the warmup and slightly raised the loss, so at this scale the results favor keeping the router on Muon.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.37914 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:23.390Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).