Metraipolis › archive › dc_9eb63072fe09

Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2 — arXiv:2610.01889

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01889
License (as archived)
CC0-1.0
Archived
2026-10-06T13:57:38.564Z
SHA-256 of the archived text
81119a3f0edac287cdf4e19ba5f07e0ae0aeafc983a6ef81da26ce2f6ae0cef4
Stable URL of this page
https://metraipolis.com/documento/dc_9eb63072fe09

How to cite

Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2 — arXiv:2610.01889 — arXiv (en), https://arxiv.org/abs/2610.01889 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9eb63072fe09 (Metraipolis archive dc_9eb63072fe09, sha256 81119a3f0edac287cdf4e19ba5f07e0ae0aeafc983a6ef81da26ce2f6ae0cef4)
BibTeX
@misc{dc9eb63072fe09,
  title = {Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2 — arXiv:2610.01889},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01889},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9eb63072fe09; sha256 81119a3f0edac287cdf4e19ba5f07e0ae0aeafc983a6ef81da26ce2f6ae0cef4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01889. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)? The answer depends on where in the network you look. We isolate this effect by holding the numerical format fixed and varying only the rounding rule at individual operation sites. To enable experiments at freely chosen precisions, we extend the PRISM vectorized rounding library to arbitrary virtual precision via a variable-precision stochastic rounding (VPSR) algorithm, proving that the rounding decision is evaluated exactly in hardware floating point. We develop two analyses providing complementary insight into this site-level trade-off. First, a probabilistic forward-error bound for linear projections shows that SR's error envelope grows as $O(\sqrt{n} u)$ in reduction length $n$, versus $O(n u)$ for RN, a gap that widens rapidly at low precision and is most pronounced in the long multilayer perceptron (MLP) down-projection. Second, a second-order decomposition of expected cross-entropy loss change at the output softmax into signed drift, drift curvature, and a Fisher-weighted variance penalty reveals why the two sites behave oppositely: MLP noise is predominantly a uniform logit shift to which softmax is invariant, so SR's variance is largely discounted; head noise is non-uniform across the vocabulary and is not. On DistilGPT-2 at $t=6$ significand bits, observations match theory: SR in the MLP raises perplexity to 1.15x the full-precision reference, versus 2.21x for RN. At the language-model head, the ordering reverses because SR introduces non-uniform variance, whereas deterministic RN carries none. In a mixed-precision configuration (MLP output at $t=6$), assigning SR to the MLP and RN to the head brings perplexity within 1.10x of the full-precision reference, a 28% reduction over matched-bit RN.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.01889 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:15:34.852Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).