Metraipolis › archive › dc_c42443e4db63
Spectrally Targeted Muon — arXiv:2610.10965
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.10965
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:21:03.615Z
- SHA-256 of the archived text
672cd6aa0ec8cf15875bfb269f2559b63c15092e7664df7069edfcf997fdf6b0- Stable URL of this page
https://metraipolis.com/documento/dc_c42443e4db63
How to cite
Spectrally Targeted Muon — arXiv:2610.10965 — arXiv (en), https://arxiv.org/abs/2610.10965 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c42443e4db63 (Metraipolis archive dc_c42443e4db63, sha256 672cd6aa0ec8cf15875bfb269f2559b63c15092e7664df7069edfcf997fdf6b0)
BibTeX
@misc{dcc42443e4db63,
title = {Spectrally Targeted Muon — arXiv:2610.10965},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.10965},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c42443e4db63; sha256 672cd6aa0ec8cf15875bfb269f2559b63c15092e7664df7069edfcf997fdf6b0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10965. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models. It remains unclear, however, whether this success comes from amplifying small singular directions that gradient descent neglects or from suppressing large, degenerate directions that disrupt training. We introduce Spectrally Targeted Muon, which orthogonalizes only the singular values above or below a threshold $\tau$, so that varying $\tau$ interpolates between normalized SGD and Muon. It isolates the relevant singular subspaces with projections computed by Newton-Schulz iteration on a shifted Gram matrix, so no SVD is needed. We evaluate these variants on the CIFAR-10 and NanoGPT speedruns, tracking the effective rank of gradient, update, and weight matrices and a new metric, the alignment of updates with the tangent space of the weight matrix's isospectral manifold. We find three things. First, the small singular values of the momentum are not noise. Orthogonalizing everything except the few largest singular values of each matrix nearly matches Muon while touching only a small fraction of the momentum, whereas orthogonalizing only the top falls well short even though it holds almost all of it. On language models every momentum singular value is far below one, so targeted orthogonalization can only amplify, and Muon wins by making directions that are too small to train on at their raw scale trainable. Second, shrinking the largest singular values is what keeps the parameter spectrum flat; this is cheap, and it is not what drives the loss. Third, AdamW differs from Muon mainly in how slowly it builds structure, which explains its slower start and why warmup helps AdamW but only hurts Muon. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.10965 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:50:31.013Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).