Metraipolis › archive › dc_80fc079d5650

Second-Moment Stochastic Approximation Methods — arXiv:2609.36600

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36600
License (as archived)
CC0-1.0
Archived
2026-10-07T16:26:11.620Z
SHA-256 of the archived text
661bb1389719a9db217805c1d1722d9b25b19f02fbfd6b76c67b4298900cfb28
Stable URL of this page
https://metraipolis.com/documento/dc_80fc079d5650

How to cite

Second-Moment Stochastic Approximation Methods — arXiv:2609.36600 — arXiv (en), https://arxiv.org/abs/2609.36600 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_80fc079d5650 (Metraipolis archive dc_80fc079d5650, sha256 661bb1389719a9db217805c1d1722d9b25b19f02fbfd6b76c67b4298900cfb28)
BibTeX
@misc{dc80fc079d5650,
  title = {Second-Moment Stochastic Approximation Methods — arXiv:2609.36600},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36600},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_80fc079d5650; sha256 661bb1389719a9db217805c1d1722d9b25b19f02fbfd6b76c67b4298900cfb28},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36600. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Classical stochastic approximation methods rely on estimators of the first moment (mean) of a random regression function. We study methods that employ estimators of both the first and the second moments, which include modern deep-learning optimizers such as Adam and Muon as special cases. We derive second-moment stochastic approximation methods through the lens of optimal preconditioning for solving matrix equations, and develop a two-stage framework for their convergence analysis. The first stage focuses on the analysis of conceptual (impractical) methods that rely on the exact first and second moments. In the second stage, we replace the exact moments with their respective estimators, and invoke Dvoretzky's theorem to show that the resulting practical methods converge almost surely to a neighborhood of the target solution. The size of the neighborhood depends on the biases and variances of the first- and second-moment estimators. We derive concrete bounds for Muon and a spectral variant of Adam that determine the radius of their neighborhood of convergence.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Optimization and Control (math.OC) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.36600 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.872Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).