Metraipolis › archive › dc_d14d3e04be2e
Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models — arXiv:2609.33290
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33290
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:52:52.607Z
- SHA-256 of the archived text
99d1382a9be5053ae8eeb86ad5cb3aaaad04dbe5448528a1ae5bc3258587f47e- Stable URL of this page
https://metraipolis.com/documento/dc_d14d3e04be2e
How to cite
Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models — arXiv:2609.33290 — arXiv (en), https://arxiv.org/abs/2609.33290 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d14d3e04be2e (Metraipolis archive dc_d14d3e04be2e, sha256 99d1382a9be5053ae8eeb86ad5cb3aaaad04dbe5448528a1ae5bc3258587f47e)
BibTeX
@misc{dcd14d3e04be2e,
title = {Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models — arXiv:2609.33290},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33290},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d14d3e04be2e; sha256 99d1382a9be5053ae8eeb86ad5cb3aaaad04dbe5448528a1ae5bc3258587f47e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33290. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning with binary correctness rewards trains correctness, not calibrated confidence. The confidence that reasoning models verbalize is systematically overconfident, and the problem is not merely one of scale: verbalized confidence tracks how willing a model is to commit to an answer, not how likely the answer is to be right. Post-hoc rescaling therefore fits one distribution but rarely transfers. We look inside the model instead. On factual question answering, a linear probe on the hidden state between the chain of thought and the answer is substantially better calibrated: its expected calibration error is 5 to 38 times lower than that of the verbalized score across four benchmarks and two model families. However, when used to pick among N sampled answers, that same probe nearly ties majority voting yet falls far short of the oracle. Internal states answer "how certain am I" well and "which answer is right" poorly, so the signal should be reported as a confidence rather than used to select answers. As a result, we introduce probe-guided self-distillation (Probe-SD): score a model's own sampled traces with the probe, overwrite the confidence each trace states, and finetune the base checkpoint of the same family, so nothing but the model itself remains at test time. On Qwen3-14B, Probe-SD cuts ECE from 0.178 to 0.024 in-domain and from 0.542 to 0.113 out-of-domain, where it also beats post-hoc recalibration and self-consistency distillation. The resulting confidence is well-calibrated and useful for weighted voting, behaviors previously attributed to online RL, here obtained with supervised finetuning alone. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.33290 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.505Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).