Metraipolis › archive › dc_dc8eba770f57
Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization — arXiv:2609.16805
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.16805
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T21:44:25.004Z
- SHA-256 of the archived text
3318cda8791b1dc22da860d3adc051904c6ad836c96fb63ab7d51e5f37dc328b- Stable URL of this page
https://metraipolis.com/documento/dc_dc8eba770f57
How to cite
Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization — arXiv:2609.16805 — arXiv (en), https://arxiv.org/abs/2609.16805 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc8eba770f57 (Metraipolis archive dc_dc8eba770f57, sha256 3318cda8791b1dc22da860d3adc051904c6ad836c96fb63ab7d51e5f37dc328b)
BibTeX
@misc{dcdc8eba770f57,
title = {Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization — arXiv:2609.16805},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.16805},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc8eba770f57; sha256 3318cda8791b1dc22da860d3adc051904c6ad836c96fb63ab7d51e5f37dc328b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16805. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit a "Ridge of Optimization" characterized by extreme stability and a highly skewed weight spectrum. However, the dynamical process by which learning converges to this critical regime has remained unclear. This paper provides a geometric analysis of the learning trajectories on the statistical manifold of a KLR-trained Hopfield network. By comparing the paths of Gradient Descent (GD) and Natural Gradient Descent (NGD), we elucidate the mechanisms governing the optimization process. Our analysis reveals that learning on the Ridge proceeds in two distinct phases. We show that the extreme curvature of the Ridge causes standard GD to follow a highly oscillatory, non-geodesic path. In stark contrast, NGD explicitly corrects for this geometry, following the ideal geodesic path and completely overcoming the instabilities faced by GD. We demonstrate experimentally that NGD not only converges significantly faster but also achieves a solution with superior generalization performance. These results establish that the highly structured geometry of the Ridge is optimally suited for information-geometric optimization, providing a new perspective on the interplay between learning dynamics and emergent representation geometry. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.NE). Categorie arXiv: Machine Learning (cs.LG) ; Neural and Evolutionary Computing (cs.NE). Pagina del lavoro: https://arxiv.org/abs/2609.16805 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T21:31:16.114Z dalla pagina https://arxiv.org/catchup/cs.NE/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).