Metraipolis › archive › dc_8a49274080da

Convergence of Practical Muon — arXiv:2609.33152

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.33152
License (as archived): CC0-1.0
Archived: 2026-10-07T22:37:13.278Z
SHA-256 of the archived text: 8999352595950741174c81650d95e2109b6a955e969b33de8273c039ed1b00c8
Stable URL of this page: https://metraipolis.com/documento/dc_8a49274080da

How to cite

Convergence of Practical Muon — arXiv:2609.33152 — arXiv (en), https://arxiv.org/abs/2609.33152 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8a49274080da (Metraipolis archive dc_8a49274080da, sha256 8999352595950741174c81650d95e2109b6a955e969b33de8273c039ed1b00c8)

BibTeX

@misc{dc8a49274080da,
  title = {Convergence of Practical Muon — arXiv:2609.33152},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33152},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8a49274080da; sha256 8999352595950741174c81650d95e2109b6a955e969b33de8273c039ed1b00c8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33152. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton--Schulz iterations with empirically tuned polynomial coefficients $(3.4445,-4.7750,2.0315)$; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted $\ell_2$ regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an $\mathcal{O}(T^{-1/4})$ convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of $\sqrt{d}$, where $T$ is the iteration horizon and $d$ is the parameter dimension. Experiments further support the theoretical convergence results.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC).
Pagina del lavoro: https://arxiv.org/abs/2609.33152 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.879Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).