Metraipolis › archive › dc_1015f37c8d2c
When Linguistic and Internal Confidence Diverge in Large Language Models — arXiv:2608.28382
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.28382
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T10:48:18.051Z
- SHA-256 of the archived text
a797d636d108eefbf2e2f41df7b7dc77b132d0b3ba0d71537b9517affbe412a4- Stable URL of this page
https://metraipolis.com/documento/dc_1015f37c8d2c
How to cite
When Linguistic and Internal Confidence Diverge in Large Language Models — arXiv:2608.28382 — arXiv (en), https://arxiv.org/abs/2608.28382 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1015f37c8d2c (Metraipolis archive dc_1015f37c8d2c, sha256 a797d636d108eefbf2e2f41df7b7dc77b132d0b3ba0d71537b9517affbe412a4)
BibTeX
@misc{dc1015f37c8d2c,
title = {When Linguistic and Internal Confidence Diverge in Large Language Models — arXiv:2608.28382},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.28382},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1015f37c8d2c; sha256 a797d636d108eefbf2e2f41df7b7dc77b132d0b3ba0d71537b9517affbe412a4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28382. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Users often ask large language models (LLMs) to report how confident they are, but it is unclear whether such linguistic confidence tracks the model's internal confidence. We study this question across 8 classification tasks, 2 generation tasks and 30 models from three families. For classification, we compare linguistic confidence with logits-based confidence along three axes: association, magnitude agreement and calibration. For generation, we test whether linguistic confidence tracks semantic-entropy-based uncertainty. The axes frequently diverge. Instance-level association is weak on average, although it improves on easier items and for stronger base models. Instruction-tuned models often report higher confidence and sometimes show higher association, but they also have larger confidence gaps and worse calibration. Prompt design mostly changes the distribution of reported confidence. Attitude cues inflate confidence without improving alignment, while score exemplars can preserve rank-order signal when they avoid collapsed confidence values. Regression analyses show that distributional properties of confidence scores explain much of the observed alignment pattern, with model metadata playing a smaller role after controls. These results support a lossy-channel view of linguistic confidence. A more dispersed verbal confidence distribution can carry useful rank information, but it does not make the scores calibrated. Linguistic confidence should therefore be evaluated with multi-axis diagnostics before being used in downstream reliability pipelines. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.28382 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T10:33:42.527Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).