Metraipolis › archive › dc_f51fdef7f596

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance — arXiv:2609.00363

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.00363
License (as archived)
CC0-1.0
Archived
2026-10-17T06:52:02.450Z
SHA-256 of the archived text
fb6315fe70c6a8e141f0e1f6b22c6eac5286074d04e4e1b25a74310fe4084087
Stable URL of this page
https://metraipolis.com/documento/dc_f51fdef7f596

How to cite

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance — arXiv:2609.00363 — arXiv (en), https://arxiv.org/abs/2609.00363 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f51fdef7f596 (Metraipolis archive dc_f51fdef7f596, sha256 fb6315fe70c6a8e141f0e1f6b22c6eac5286074d04e4e1b25a74310fe4084087)
BibTeX
@misc{dcf51fdef7f596,
  title = {Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance — arXiv:2609.00363},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.00363},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f51fdef7f596; sha256 fb6315fe70c6a8e141f0e1f6b22c6eac5286074d04e4e1b25a74310fe4084087},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00363. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance. We measure what such a suite can detect. Injecting nine faults into a reference INT8 pipeline over 8,232 layer--fault--regime cells of Qwen3-1.7B, we find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing, and by exactly one whenever it moves it at all, across 5,880 cells. A tolerance of one spacing is therefore blind to the entire class by construction: four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales. Faults that violate the accumulator's exactness preconditions, or that break operand sharing, are detected without exception, and a null fault never fires. What a tolerance-based suite of this shape establishes is therefore narrower than interchangeability: that the preconditions hold, that operands are shared, and that differences stay within one spacing. The power-of-two constraint that exposes the one detected fault is also deployable. Requantizing every weight scale to its nearest power of two makes CUTLASS and Triton agree bitwise at every linear layer (196/196 and 252/252, against 8/196 and 10/252 under the checkpoints' own scales) and yields byte-identical generated token sequences at 1.7B, 8B and 14B (8/8 prompts, against 0/8 at all three). Observed perplexity point estimates are +0.32%, -0.28% and +0.48%; the 90% intervals cover zero at the two smaller sizes but not at 14B, reaching +0.71% and +0.76%. A previously reported +157% perplexity for this intervention was an artifact of a probe that rewrote scales without requantizing the weights; separating the effects attributes 99.8% of it to the resulting weight--scale mismatch rather than to the power-of-two constraint itself.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.00363 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:41:55.774Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).