Metraipolis › archive › dc_8779b84f1423

The Best Optimizer Depends on Batch Size — arXiv:2610.08975

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08975
License (as archived)
CC0-1.0
Archived
2026-10-08T11:56:09.838Z
SHA-256 of the archived text
a9ff3a51bf799629de3773ce5dc2d3da8a0c5efd16dd4b25f65d778dad2d2d67
Stable URL of this page
https://metraipolis.com/documento/dc_8779b84f1423

How to cite

The Best Optimizer Depends on Batch Size — arXiv:2610.08975 — arXiv (en), https://arxiv.org/abs/2610.08975 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8779b84f1423 (Metraipolis archive dc_8779b84f1423, sha256 a9ff3a51bf799629de3773ce5dc2d3da8a0c5efd16dd4b25f65d778dad2d2d67)
BibTeX
@misc{dc8779b84f1423,
  title = {The Best Optimizer Depends on Batch Size — arXiv:2610.08975},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08975},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8779b84f1423; sha256 a9ff3a51bf799629de3773ce5dc2d3da8a0c5efd16dd4b25f65d778dad2d2d67},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08975. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A plethora of new adaptive optimizers are designed to efficiently estimate and use minibatch gradient statistics to shape parameter updates, but they are typically benchmarked at a single batch size. Hyperparameter scaling rules promise to preserve performance as batch size and gradient noise change, suggesting that the best optimizer at one batch size should remain the best at another. We challenge this approach to developing and evaluating optimizers by showing: (1) no principled scaling rule for Muon works consistently across training settings, and (2) the best optimizer for language model pretraining changes with batch size even after extensive hyperparameter tuning.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.08975 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:50:37.633Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).