Metraipolis › archive › dc_38097ca2880d
Beyond Confidence: Rethinking Uncertainty Calibration in Deep Code Models — arXiv:2610.21248
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.21248
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T10:25:27.332Z
- SHA-256 of the archived text
c051da905465e289c8887f9403b7d69de13c4f12e758868b024f7b805631bd5c- Stable URL of this page
https://metraipolis.com/documento/dc_38097ca2880d
How to cite
Beyond Confidence: Rethinking Uncertainty Calibration in Deep Code Models — arXiv:2610.21248 — arXiv (en), https://arxiv.org/abs/2610.21248 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_38097ca2880d (Metraipolis archive dc_38097ca2880d, sha256 c051da905465e289c8887f9403b7d69de13c4f12e758868b024f7b805631bd5c)
BibTeX
@misc{dc38097ca2880d,
title = {Beyond Confidence: Rethinking Uncertainty Calibration in Deep Code Models — arXiv:2610.21248},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.21248},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_38097ca2880d; sha256 c051da905465e289c8887f9403b7d69de13c4f12e758868b024f7b805631bd5c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21248. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Deep learning models have shown significant promise in code-related tasks. While confidence calibration techniques have been proposed to improve model reliability, their effectiveness in real-world tasks with strict accuracy requirements remains uncertain. In this paper, we investigate two key questions: first, whether existing measurement metrics, including confidence scores and uncertainty metrics, are effective for guiding selective prediction; and second, whether existing calibration techniques can improve these metrics to better identify cases where model predictions are likely to succeed or fail. We conduct a systematic study on defect prediction and vulnerability detection using seven code models, evaluating thirteen uncertainty signals and three widely used post-hoc calibration methods. Our analysis reveals three major findings: (1) even strong uncertainty signals can remain poorly calibrated and provide limited separation at the high-confidence end, constraining selective prediction in practice; (2) standard post-hoc calibrators often improve calibration scores but do not consistently translate into better selective prediction behavior; and (3) commonly used calibration metrics can disagree with selective prediction performance, underscoring the need to evaluate both jointly. Moreover, our findings motivate a decision-oriented view of reliability in deep code models, where calibrated scores must be evaluated together with selective-prediction quality and retained-region risk before they can support accept--defer deployment. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2610.21248 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:48:17.697Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).