Metraipolis › archive › dc_8895254bf02b

Exact information accounting for SGD methods — arXiv:2610.00446

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00446
License (as archived)
CC0-1.0
Archived
2026-10-06T15:33:25.878Z
SHA-256 of the archived text
8c3d1b2dc8850e2afe83c1686cdb2e1b7ebfb71ee2fb8b15bd6e4e1362decf0e
Stable URL of this page
https://metraipolis.com/documento/dc_8895254bf02b

How to cite

Exact information accounting for SGD methods — arXiv:2610.00446 — arXiv (en), https://arxiv.org/abs/2610.00446 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8895254bf02b (Metraipolis archive dc_8895254bf02b, sha256 8c3d1b2dc8850e2afe83c1686cdb2e1b7ebfb71ee2fb8b15bd6e4e1362decf0e)
BibTeX
@misc{dc8895254bf02b,
  title = {Exact information accounting for SGD methods — arXiv:2610.00446},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00446},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8895254bf02b; sha256 8c3d1b2dc8850e2afe83c1686cdb2e1b7ebfb71ee2fb8b15bd6e4e1362decf0e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00446. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As an alternative to the standard geometric analyses, we give an exact, information-theoretic analysis of stochastic gradient descent (SGD) and its variants. We show that a preconditioned SGD step is the posterior-mean update of a Gaussian Bayes model, and that its one-step regret splits into an intrinsic-time cost and a change in comparator information. The split extends to an identity for the objective itself. Convex convergence, strict-saddle-point escape, the link between flatness and generalization, the standard learning-rate schedules, adaptive optimizers, and the noisy, momentum, heavy-tailed, and gradient-free variants of SGD each correspond to a term or a special case of this identity. We measure its terms on synthetic and real training runs. On real networks it attributes the slack of classical convergence bounds to the terms their derivations drop and separates optimizers that reach the same training loss. That separation follows the number and consistency of their steps. Its relation to which of them generalizes better differs between networks. For gradient-free SGD the identity determines how a curvature preconditioner should enter the update. The sharpness-based generalization certificate it yields, with a data-independent isotropic prior, is vacuous at network scale unless the curvature spectrum is nearly flat across all parameters.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-10-02 in stat.ML).
Categorie arXiv: Machine Learning (cs.LG) ; Information Theory (cs.IT); Optimization and Control (math.OC); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.00446 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T15:12:55.888Z dalla pagina https://arxiv.org/catchup/stat.ML/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).