Metraipolis › archive › dc_f7b89aa5fe10
How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( \sqrt{ \log d } ) Divergence — arXiv:2610.07228
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07228
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:42:17.612Z
- SHA-256 of the archived text
38ecdb3fffefaaa43762c18707302c93accf30eae03a32b44f596ba56a304434- Stable URL of this page
https://metraipolis.com/documento/dc_f7b89aa5fe10
How to cite
How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( \sqrt{ \log d } ) Divergence — arXiv:2610.07228 — arXiv (en), https://arxiv.org/abs/2610.07228 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f7b89aa5fe10 (Metraipolis archive dc_f7b89aa5fe10, sha256 38ecdb3fffefaaa43762c18707302c93accf30eae03a32b44f596ba56a304434)
BibTeX
@misc{dcf7b89aa5fe10,
title = {How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( sqrt log d ) Divergence — arXiv:2610.07228},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07228},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f7b89aa5fe10; sha256 38ecdb3fffefaaa43762c18707302c93accf30eae03a32b44f596ba56a304434},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07228. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Natural gradient descent (NGD) underlies common methods in ML. For dually flat families, idealized NGD on the forward Kullback--Leibler objective follows the mixture geodesic which is often longer than the shortest Fisher--Rao path. We quantify this overhead by the inefficiency ratio \(R \ge 1\), the Fisher length of the mixture geodesic divided by the Fisher--Rao distance, and bound its supremum over endpoint pairs as a function of the parameter dimension \(d\). A tensor criterion identifies the regime (I) families, with \(R=1\) everywhere: exactly those with quadratic potential or dimension one, such as fixed-covariance Gaussians. For non-quadratic families, we prove two further regimes: (II) bounded third-order skewness plus finite Fisher--Rao diameter yields a dimension-independent bound; and (III) for products of scale families---including Gaussian covariances and Gamma rates---\(R\) grows as \(\Theta(\sqrt{\log d})\), unbounded in \(d\). Under a per-step Fisher-chord budget, \(R\) translates to a practical computational cost: NGD requires asymptotically at least \(R\) times as many steps as an optimizer following the Fisher--Rao geodesic. Experiments confirm all three regimes: \(R=1\) to machine precision for quadratic-potential families (I), the categorical bound \(\pi/(2\sqrt{2})\) is approached but not attained (II), and sampled scale-product \(R\) grows with \(d\), reaching \(R \approx 1.5\) for long, high-dimensional moves (III).
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.07228 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:19:24.881Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).