Metraipolis › archive › dc_98309733baeb

A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality — arXiv:2609.18005

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18005
License (as archived)
CC0-1.0
Archived
2026-10-15T13:49:18.462Z
SHA-256 of the archived text
44610007f33eae57b0e17933363219633fa49481d2fbd98b47d0c47a6cc08f3d
Stable URL of this page
https://metraipolis.com/documento/dc_98309733baeb

How to cite

A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality — arXiv:2609.18005 — arXiv (en), https://arxiv.org/abs/2609.18005 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_98309733baeb (Metraipolis archive dc_98309733baeb, sha256 44610007f33eae57b0e17933363219633fa49481d2fbd98b47d0c47a6cc08f3d)
BibTeX
@misc{dc98309733baeb,
  title = {A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality — arXiv:2609.18005},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18005},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_98309733baeb; sha256 44610007f33eae57b0e17933363219633fa49481d2fbd98b47d0c47a6cc08f3d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18005. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model optimization is an active research area, spanning quantization of model weights, early-exit methods for skipping layers, and speculative decoding. Each track uses its own quality measures, typically an idiosyncratic benchmark score. Few approach the measurement precision required by other scientific disciplines. We propose a rigorous methodology for measuring output quality, suitable for cross-system and cross-technique comparison. We score outputs with an LLM as a judge, but calibrate the judge formally: we compare its scores on two ordinary runs of a model given the same prompts, verifying that it shows no systematic preference between statistically equivalent outputs and measuring its per-sample noise. Each design also includes a 'null' condition, provably identical in distribution to the unmodified model, whose measured difference must be zero. With this one instrument we measure several acceleration techniques on the same prompts, so their quality costs can be compared. Perceived quality proves highly dependent on the domain of discourse. A 4-bit model was indistinguishable from its 16-bit original down to our design's +/-0.3-point resolution, in English prose and Chinese alike. At 3-bit precision the same prompts lost 0.5 points in English prose, 0.9 in Chinese, and 1.1 on multi-step math; early exit that cost 0.7 points on prose cost 2.5 on math, cutting correctly solved problems from 19 of 27 to 6. The pattern held for models from Alibaba and from Meta, but not its magnitude: the same quantizer cost Meta's model 1.8 points where it cost Alibaba's 0.7. A model's certainty about a token predicts how likely it is to differ from the full model's choice, but not how much that difference affects judged quality, so acceptance rules relying on certainty cannot distinguish errors that matter from errors that don't.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.18005 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:48:24.290Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).