Metraipolis › archive › dc_d7f2b9fb79bd
ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training — arXiv:2610.04140
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.04140
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:28:42.470Z
- SHA-256 of the archived text
a2426743e5cc9f04f94ec89897841b7d3cfbb42d3290110c846f810f9d26d5aa- Stable URL of this page
https://metraipolis.com/documento/dc_d7f2b9fb79bd
How to cite
ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training — arXiv:2610.04140 — arXiv, https://arxiv.org/abs/2610.04140 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d7f2b9fb79bd (Metraipolis archive dc_d7f2b9fb79bd, sha256 a2426743e5cc9f04f94ec89897841b7d3cfbb42d3290110c846f810f9d26d5aa)
BibTeX
@misc{dcd7f2b9fb79bd,
title = {ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training — arXiv:2610.04140},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.04140},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d7f2b9fb79bd; sha256 a2426743e5cc9f04f94ec89897841b7d3cfbb42d3290110c846f810f9d26d5aa},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04140. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-experts (MoE) language models send each token to a few experts, so each expert is trained on a different part of the data, and this part changes during training. With a shared learning rate, Muon applies updates of roughly the same size to expert matrices of the same shape, even when an expert's update is poorly aligned with its current gradient. We here propose ExpertMuon-Compass (Compass), which multiplies the Muon step of each expert by two factors. A family factor compares the cosine between the expert's orthogonalized update and its gradient with the same cosine for the other experts in its layer. A scalar radius aggregates the alignment between corresponding rows of the update and gradient into one step-length multiplier. Compass keeps the update direction and the momentum buffer of Muon. In pretraining on FineWeb-Edu, Compass with Nesterov momentum on all matrices performs as well as or better than Muon, NorMuon, and other optimizers, with weight decay matched to NorMuon in the longer runs. Adding its factors to NorMuon gives the same or a lower loss than NorMuon. Compass is the most effective when the data seen by each expert varies during training, for example, as when the languages of a multilingual corpus arrive in separate blocks. With Compass, the expert load stays balanced, and the router assigns tokens to experts more decisively. We also prove a perturbation bound for the two factors. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Optimization and Control (math.OC). Pagina del lavoro: https://arxiv.org/abs/2610.04140 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:20.401Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).