Metraipolis › archive › dc_2bdbae057215

When and How Does Muon Precondition? The Role of Structured Gradient Noise — arXiv:2610.31728

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.31728
License (as archived)
CC0-1.0
Archived
2026-10-26T20:11:16.390Z
SHA-256 of the archived text
25cc77de46bc3881b1106396564a8fcb2961b042beadeae873f93be76a518d51
Stable URL of this page
https://metraipolis.com/documento/dc_2bdbae057215

How to cite

When and How Does Muon Precondition? The Role of Structured Gradient Noise — arXiv:2610.31728 — arXiv (en), https://arxiv.org/abs/2610.31728 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2bdbae057215 (Metraipolis archive dc_2bdbae057215, sha256 25cc77de46bc3881b1106396564a8fcb2961b042beadeae873f93be76a518d51)
BibTeX
@misc{dc2bdbae057215,
  title = {When and How Does Muon Precondition? The Role of Structured Gradient Noise — arXiv:2610.31728},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31728},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2bdbae057215; sha256 25cc77de46bc3881b1106396564a8fcb2961b042beadeae873f93be76a518d51},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31728. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon's preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate. This motivates us to investigate structured noise arising from mini-batch sampling in associative memory with heterogeneous task frequencies. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning. Our analysis shows that Muon can soften the dependence on task frequency from linear to square-root, helping balance the learning of frequent and rare tasks. We further identify rotation-induced negative transfer in SignSGD that Muon avoids, and explain how momentum reduces sampling noise while tracking changing gradients. Controlled numerical experiments validate these mechanisms, and a preliminary LLM pretraining study illustrates the practical promise of fast-slow momentum.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.31728 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:49:03.353Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).