Metraipolis › archive › dc_ef02f87bd7cf

Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41074
License (as archived): CC0-1.0
Archived: 2026-11-03T07:52:25.145Z
SHA-256 of the archived text: a24b1b47f2af32cb4066c19b93c5eeb5fa99e81d56634a152a6233fffb162220
Stable URL of this page: https://metraipolis.com/documento/dc_ef02f87bd7cf

How to cite

Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074 — arXiv (en), https://arxiv.org/abs/2610.41074 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef02f87bd7cf (Metraipolis archive dc_ef02f87bd7cf, sha256 a24b1b47f2af32cb4066c19b93c5eeb5fa99e81d56634a152a6233fffb162220)

BibTeX

@misc{dcef02f87bd7cf,
  title = {Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41074},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef02f87bd7cf; sha256 a24b1b47f2af32cb4066c19b93c5eeb5fa99e81d56634a152a6233fffb162220},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41074. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-precision training reduces computation but introduces additional rounding error. We theoretically study how stochastic rounding (SR) affects prediction risk as model size, training time, and precision change. We consider a two-layer linear network with a fixed random first layer, trained by mini-batch SGD on fresh Gaussian data with power-law spectra. At each SGD step, the features used in forward and backward computations are rounded independently. We prove matched upper and lower bounds on prediction error for the final model and for uniform parameter averaging, with general learning-rate and precision schedules chosen before training. For fixed precisions and constant learning rates with parameter averaging, we derive how the smallest prediction error decreases with compute, and we identify model sizes and training times that achieve this optimal-compute scaling. Rounding only the forward features preserves the compute scaling of exact arithmetic, while rounding backward features changes it. Averaging parameters can change which computation needs higher precision and when higher precision is useful. Our results also give bit requirements for a given accuracy tolerance and show how backward rounding can change learning-rate schedule design. At a fixed compute budget, lower precision can improve prediction when the benefit of a larger model or longer training outweighs the additional SR-induced error. Model simulations illustrate these tradeoffs, and experiments with an MLP and a small transformer examine rounding choices beyond the theoretical model.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.41074 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:49:31.133Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).