Metraipolis › archive › dc_cde10f8d5def
Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization — arXiv:2610.04249
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.04249
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:30:28.465Z
- SHA-256 of the archived text
119a3d3ee4dc14b612e0bf9cfb5041b7979c3044e2dc539ea1bcc7bd7768b8f8- Stable URL of this page
https://metraipolis.com/documento/dc_cde10f8d5def
How to cite
Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization — arXiv:2610.04249 — arXiv, https://arxiv.org/abs/2610.04249 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cde10f8d5def (Metraipolis archive dc_cde10f8d5def, sha256 119a3d3ee4dc14b612e0bf9cfb5041b7979c3044e2dc539ea1bcc7bd7768b8f8)
BibTeX
@misc{dccde10f8d5def,
title = {Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization — arXiv:2610.04249},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.04249},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cde10f8d5def; sha256 119a3d3ee4dc14b612e0bf9cfb5041b7979c3044e2dc539ea1bcc7bd7768b8f8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04249. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing how different components progress during training. In deep matrix factorization, the geometry that slows gradient descent (GD) favors low-rank solutions by delaying the emergence of small singular modes. This raises the question of what remains of that spectral bias when normalization or curvature correction weakens or removes the slowdown. We study how optimizer geometry and depth jointly govern this behavior through a common framework for singular mode dynamics. Under explicit balance and alignment assumptions, we derive birth, saturation, and decay laws for Euclidean GD, coordinate-wise updates of SignGD and an instantaneous Adam approximation, spectral updates of Muon and cumulative Shampoo, and a block curvature model of K-FAC. The resulting picture is not a simple ordering from stronger to weaker low-rank bias. To highlight, SignGD and ideal Muon eliminate the divergent birth barrier and drive unsupported modes to zero in finite time. Cumulative Shampoo initially retains GD's depth-dependent barrier, then accumulated gradients produce a catch-up phase while making previously active modes increasingly persistent. Undamped K-FAC cancels the factorization-induced slowdown while preserving the ordering of the target singular values, whereas positive damping introduces a spectral threshold below which the slow GD phase laws reappear. These results give normalization, accumulated state, damping, and depth a direct interpretation as controls determining when modes emerge, persist, and disappear during training. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.04249 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:27:05.666Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).