Metraipolis › archive › dc_2bda05ed322f
Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41074
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-02T04:44:35.473Z
- SHA-256 of the archived text
a893784e075fe8273a8a2ac64286a1adfe24b86c75c6fb66ccdcc29e866ddea9- Stable URL of this page
https://metraipolis.com/documento/dc_2bda05ed322f
How to cite
Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074 — arXiv (en), https://arxiv.org/abs/2610.41074 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2bda05ed322f (Metraipolis archive dc_2bda05ed322f, sha256 a893784e075fe8273a8a2ac64286a1adfe24b86c75c6fb66ccdcc29e866ddea9)
BibTeX
@misc{dc2bda05ed322f,
title = {Where, When, and How Many Bits? Scaling Laws for SGD with Stochastic Rounding — arXiv:2610.41074},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41074},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2bda05ed322f; sha256 a893784e075fe8273a8a2ac64286a1adfe24b86c75c6fb66ccdcc29e866ddea9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41074. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Low-precision training reduces computation but introduces additional rounding error. We theoretically study how stochastic rounding (SR) affects prediction risk as model size, training time, and precision change. We consider a two-layer linear network with a fixed random first layer, trained by mini-batch SGD on fresh Gaussian data with power-law spectra. At each SGD step, the features used in forward and backward computations are rounded independently. We prove matched upper and lower bounds on prediction error for the final model and for uniform parameter averaging, with general learning-rate and precision schedules chosen before training. For fixed precisions and constant learning rates with parameter averaging, we derive how the smallest prediction error decreases with compute, and we identify model sizes and training times that achieve this optimal-compute scaling. Rounding only the forward features preserves the compute scaling of exact arithmetic, while rounding backward features changes it. Averaging parameters can change which computation needs higher precision and when higher precision is useful. Our results also give bit requirements for a given accuracy tolerance and show how backward rounding can change learning-rate schedule design. At a fixed compute budget, lower precision can improve prediction when the benefit of a larger model or longer training outweighs the additional SR-induced error. Model simulations illustrate these tradeoffs, and experiments with an MLP and a small transformer examine rounding choices beyond the theoretical model. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2610.41074 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-02T04:42:12.446Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).