Metraipolis › archive › dc_48bf7553b12e

Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41074
License (as archived)
CC0-1.0
Archived
2026-11-03T05:32:46.576Z
SHA-256 of the archived text
2e5323f6c3145ce182eda19446f71245bc4075bc3b114348bd1c6a222ada4bde
Stable URL of this page
https://metraipolis.com/documento/dc_48bf7553b12e

How to cite

Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074 — arXiv (en), https://arxiv.org/abs/2610.41074 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_48bf7553b12e (Metraipolis archive dc_48bf7553b12e, sha256 2e5323f6c3145ce182eda19446f71245bc4075bc3b114348bd1c6a222ada4bde)
BibTeX
@misc{dc48bf7553b12e,
  title = {Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41074},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_48bf7553b12e; sha256 2e5323f6c3145ce182eda19446f71245bc4075bc3b114348bd1c6a222ada4bde},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41074. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-precision training reduces computation but introduces additional rounding error. We theoretically study how stochastic rounding (SR) affects prediction risk as model size, training time, and precision change. We consider a two-layer linear network with a fixed random first layer, trained by mini-batch SGD on fresh Gaussian data with power-law spectra. At each SGD step, the features used in forward and backward computations are rounded independently. We prove matched upper and lower bounds on prediction error for the final model and for uniform parameter averaging, with general learning-rate and precision schedules chosen before training. For fixed precisions and constant learning rates with parameter averaging, we derive how the smallest prediction error decreases with compute, and we identify model sizes and training times that achieve this optimal-compute scaling. Rounding only the forward features preserves the compute scaling of exact arithmetic, while rounding backward features changes it. Averaging parameters can change which computation needs higher precision and when higher precision is useful. Our results also give bit requirements for a given accuracy tolerance and show how backward rounding can change learning-rate schedule design. At a fixed compute budget, lower precision can improve prediction when the benefit of a larger model or longer training outweighs the additional SR-induced error. Model simulations illustrate these tradeoffs, and experiments with an MLP and a small transformer examine rounding choices beyond the theoretical model.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.41074 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T05:18:52.193Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).