Metraipolis › archive › dc_dbd372618980

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence — arXiv:2610.22139

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22139
License (as archived)
CC0-1.0
Archived
2026-10-16T09:05:23.829Z
SHA-256 of the archived text
e932c80d0f4f020266e6ceb66ad3c23cc41eb67f8c012559af1046294b8cec7b
Stable URL of this page
https://metraipolis.com/documento/dc_dbd372618980

How to cite

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence — arXiv:2610.22139 — arXiv (en), https://arxiv.org/abs/2610.22139 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dbd372618980 (Metraipolis archive dc_dbd372618980, sha256 e932c80d0f4f020266e6ceb66ad3c23cc41eb67f8c012559af1046294b8cec7b)
BibTeX
@misc{dcdbd372618980,
  title = {Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence — arXiv:2610.22139},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22139},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dbd372618980; sha256 e932c80d0f4f020266e6ceb66ad3c23cc41eb67f8c012559af1046294b8cec7b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22139. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Confidence scores determine whether an LLM answer is accepted, reviewed, or regenerated, yet scalar recalibration can only correct errors visible through the score itself. We introduce RADAR (Repair, Adopt, and Diagnose After Recalibration), a reliability framework that separates confidence repair from deciding whether to deploy the repair and auditing the errors that remain. Under squared loss, we decompose prediction risk into outcome noise, information hidden by the score, and mapping error; restricting repair to order-preserving maps additionally introduces a pooling gap. For repair, we propose Probe-Anchored Isotonic Calibration (PAIC), which learns a monotone map from correctness-probe predictions on unlabeled target prompts. PAIC's excess risk depends on the probe's score-conditional bias and fitting error rather than its overall accuracy. For adoption, we derive a paired procedure with finite-sample control of false improvements. For auditing, we develop a prompt-clustered kernel test that detects feature-aligned residual errors while accounting for dependent answers from the same prompt. Across multiple LLM checkpoints, benchmarks, and distribution shifts, richer predictors can improve confidence estimation, but projecting their information back to a scalar score can lose these gains, and lower prediction risk need not yield better review decisions. Across 47,400 replayed raw-score allocations, paired betting adopts candidates with lower held-out test risk in 42.95% of allocations, versus 15.09% for one-sided empirical Bernstein. RADAR thus provides a principled workflow to repair what a scalar score can express, adopt a repair only when supported by evidence, and diagnose systematic errors that recalibration leaves behind. Code and data will be released upon acceptance.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.22139 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:17.816Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).