Metraipolis › archive › dc_4798c1946d3b

Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima — arXiv:2609.32861

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.32861
License (as archived): CC0-1.0
Archived: 2026-10-07T22:36:55.177Z
SHA-256 of the archived text: 59b9fd6c71521f8e74ec1ba3e893ade7f8f2331ea8803da17e37ab84c2343c69
Stable URL of this page: https://metraipolis.com/documento/dc_4798c1946d3b

How to cite

Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima — arXiv:2609.32861 — arXiv (en), https://arxiv.org/abs/2609.32861 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4798c1946d3b (Metraipolis archive dc_4798c1946d3b, sha256 59b9fd6c71521f8e74ec1ba3e893ade7f8f2331ea8803da17e37ab84c2343c69)

BibTeX

@misc{dc4798c1946d3b,
  title = {Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima — arXiv:2609.32861},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32861},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4798c1946d3b; sha256 59b9fd6c71521f8e74ec1ba3e893ade7f8f2331ea8803da17e37ab84c2343c69},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32861. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Muon replaces the momentum buffer of each weight matrix by its orthogonal polar factor. We ask what this orthogonalization does near an optimum, where minibatch noise dominates the gradient. Under a Gaussian noise model, the expected Muon update becomes a scaled gradient step, so a linear method with a suitably matched learning rate reproduces Muon's first-order mean response. The stochastic update is a different matter: after the response is matched, Muon retains a nonlinear residual that is uncorrelated with the input noise and contributes additional covariance. A Hermite expansion shows how momentum acts on this residual. Its higher-order components decorrelate faster than the linear component, so momentum suppresses the residual's accumulated covariance relative to the linear part, but never eliminates it. In a local quadratic surrogate that evaluates the residual on the stationary noise buffer, the residual adds stationary covariance and raises the stationary loss floor at every stable step size, while leaving the contraction dynamics unchanged. Simulations of the full nonlinear recursion on quadratics and measurements on frozen transformer gradients support each step of this picture. Together, the results make a theoretical case for replacing orthogonalization by response-matched momentum SGD once optimization becomes noise-dominated.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32861 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.789Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).