Metraipolis › archive › dc_201b1db7ea5d
Calibration as a First-Class Criterion in LLM Evaluation — arXiv:2609.26489
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.26489
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T14:36:48.635Z
- SHA-256 of the archived text
4408010b192a8e8a03901a913a1bcc5def216e673714ea9200df33a4ff6f306e- Stable URL of this page
https://metraipolis.com/documento/dc_201b1db7ea5d
How to cite
Calibration as a First-Class Criterion in LLM Evaluation — arXiv:2609.26489 — arXiv (en), https://arxiv.org/abs/2609.26489 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_201b1db7ea5d (Metraipolis archive dc_201b1db7ea5d, sha256 4408010b192a8e8a03901a913a1bcc5def216e673714ea9200df33a4ff6f306e)
BibTeX
@misc{dc201b1db7ea5d,
title = {Calibration as a First-Class Criterion in LLM Evaluation — arXiv:2609.26489},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.26489},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_201b1db7ea5d; sha256 4408010b192a8e8a03901a913a1bcc5def216e673714ea9200df33a4ff6f306e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26489. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it already exist. The problem is adoption: outside this subfield, NLP research regularly introduces new models, datasets, and benchmarks without checking whether the model's confidence scores are meaningful. We argue that this adoption gap is a major obstacle to trustworthy LLM evaluation. Miscalibration causes problems in two distinct areas: at deployment, where overconfident mistakes cause real harm, and inside the research pipeline, where methods like LLM-as-a-judge, synthetic data generation, and active learning rely on calibrated confidence without verifying it. Standard calibration metrics only require two inputs per example: a confidence score and a correctness judgment. Most benchmarks in use today already provide both, meaning calibration can be reported immediately. For open-ended generation, however, defining these two inputs is still an open challenge. We argue that each NLP subfield should pair its main performance metric with a calibration score and call for treating calibration as an essential property of every model rather than a niche topic. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.26489 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T13:46:15.756Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).