Metraipolis › archive › dc_4170ef6c87fc

How Bregman Divergences Shape Shampoo — arXiv:2610.08534

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08534
License (as archived)
CC0-1.0
Archived
2026-10-07T11:41:50.701Z
SHA-256 of the archived text
0dfd1a95b05bf3280980410feaedcfdc433f1b5eaa0e809435555a61a6987793
Stable URL of this page
https://metraipolis.com/documento/dc_4170ef6c87fc

How to cite

How Bregman Divergences Shape Shampoo — arXiv:2610.08534 — arXiv (en), https://arxiv.org/abs/2610.08534 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4170ef6c87fc (Metraipolis archive dc_4170ef6c87fc, sha256 0dfd1a95b05bf3280980410feaedcfdc433f1b5eaa0e809435555a61a6987793)
BibTeX
@misc{dc4170ef6c87fc,
  title = {How Bregman Divergences Shape Shampoo — arXiv:2610.08534},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08534},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4170ef6c87fc; sha256 0dfd1a95b05bf3280980410feaedcfdc433f1b5eaa0e809435555a61a6987793},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08534. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Understanding the principles behind Shampoo has recently guided the development of more effective neural network optimizers. These methods learn a preconditioner by optimizing the Frobenius or Kullback-Leibler (KL) divergence against the gradient second moment. In this work, we investigate how the choice of divergence shapes preconditioning, which remains unclear and blocks further improvements. To do so, we develop a unified Bregman divergence framework that connects all popular divergences, allowing us to study them jointly. Through empirical spectral analysis of gradient second moments, we examine how divergence choice shapes Kronecker approximation and interacts with finite-sample error in preconditioning. We find that some divergences can better compensate for finite-sample underestimation of the empirical second moment, helping explain the differing behavior of their corresponding Shampoo variants. We further validate this explanation through GPT-2 pretraining experiments. By connecting divergence choice to practical training behavior, we believe our framework provides principled guidance for understanding the foundations of, and further improving, Shampoo.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.08534 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:19:24.835Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).