Metraipolis › archive › dc_1d09fbd256ec
Beyond the Matrix Sign: Quadratic Spectral Descent — arXiv:2609.07597
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.07597
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:57:50.735Z
- SHA-256 of the archived text
39df1cd0817002314e77c708c8a5dc74fed8d0b1ef252ae792975cedc3406bba- Stable URL of this page
https://metraipolis.com/documento/dc_1d09fbd256ec
How to cite
Beyond the Matrix Sign: Quadratic Spectral Descent — arXiv:2609.07597 — arXiv (en), https://arxiv.org/abs/2609.07597 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1d09fbd256ec (Metraipolis archive dc_1d09fbd256ec, sha256 39df1cd0817002314e77c708c8a5dc74fed8d0b1ef252ae792975cedc3406bba)
BibTeX
@misc{dc1d09fbd256ec,
title = {Beyond the Matrix Sign: Quadratic Spectral Descent — arXiv:2609.07597},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.07597},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1d09fbd256ec; sha256 39df1cd0817002314e77c708c8a5dc74fed8d0b1ef252ae792975cedc3406bba},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07597. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Muon emerges as a strong competitor of the AdamW for LLM pretraining, because the matrix-wise update it employs can potentially incur smaller second-order penalty than the once dominating AdamW, which performs coordinate-wise update. However, the spectral flattening procedure in Muon is quite debatable since it discards the spectral amplitude information totally. To seek for better spectral allocation (and the associated spectral subspace), we propose to solve the quadratic model of loss function under the spectral norm constraint \textit{directly} (i.e., in a genuinely Newtonian way) and thus obtaining the Quadratic Spectral Descent (QSD) algorithm. In contrast, many existing curvature-aware methods either exploit the second-order information in an \textit{implicit} way by changing the weight update geometry (such as Mousse, FISMO) or rely on strong assumptions (such as the weight displacement isotropy assumption in Newton-Muon). QSD's potential advantage over these methods is best illustrated in the isotropic curvature scenario, where Mousse, FISMO and Newton-Muon all reduce to Muon while the spectral allocation in QSD is still \textit{non-flat} (since the spectral allocation in QSD depends on the \textit{gradient to curvature ratio}). Meanwhile, to control the complexity of QSD, we employ inversion-free K-FAC and \textit{online} Frank-Wolfe update which is essentially a matrix sign operator. Overall, the complexity increase can be rather mild. Experiments on GPT pre-training show that QSD consistently improves validation loss over Muon and recent Muon variants, while achieving up to an $8.49\%$ wall-clock speedup at matched validation loss.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.07597 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:42.967Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).