Metraipolis › archive › dc_5a19e292cccd
Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model — arXiv:2610.01006
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.01006
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T12:03:27.181Z
- SHA-256 of the archived text
72cfe8753d230272c3ea1b26bbde49a8f30f6b738340e67ec64881b4545550a5- Stable URL of this page
https://metraipolis.com/documento/dc_5a19e292cccd
How to cite
Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model — arXiv:2610.01006 — arXiv (en), https://arxiv.org/abs/2610.01006 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5a19e292cccd (Metraipolis archive dc_5a19e292cccd, sha256 72cfe8753d230272c3ea1b26bbde49a8f30f6b738340e67ec64881b4545550a5)
BibTeX
@misc{dc5a19e292cccd,
title = {Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model — arXiv:2610.01006},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.01006},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5a19e292cccd; sha256 72cfe8753d230272c3ea1b26bbde49a8f30f6b738340e67ec64881b4545550a5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01006. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Decision models return probabilities intended for routing, abstention and automated action. Calibration makes those probabilities useful on average, but does not establish whether low confidence reflects chance or missing knowledge, nor whether confidence falls when a model moves beyond what it knows. We audit this distinction in Jev, a decision model, with over 15 public datasets and 6 generated task families, with paired interventions that vary the information supplied for a fixed item. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge: with no answer-relevant information it assigns up to 0.80 to a salient option, and on news beyond an observed knowledge boundary it exceeds accuracy by 0.21--0.33, a gap that recalibration on earlier months does not close. Targeted yes/no questions give sharper readouts of the case: whether an outcome is settled (AUROC 1.00) and whether the evidence suffices (0.95, against 0.85 for confidence on the same items). Asking whether Jev knows the answer appears to flag fabricated entities and post-boundary news (0.91), but with realistic names or with dates removed it shows no advantage over answer uncertainty. Black-box knowledge audits therefore need explicit controls for surface cues. Code: this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.01006 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T10:21:27.904Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).