Metraipolis › archive › dc_bfa605bd610d
arXiv 2610.26645: Reasoning-Aware Multicalibrated Reliability Predictors for Language Models
- Source
- Metraipolis archive
- Original
- https://arxiv.org/abs/2610.26645
- License (as archived)
- CC-BY-4.0
- Archived
- 2026-10-20T15:28:05.750Z
- SHA-256 of the archived text
b7563752b05283ade3b87879490c29a492b501b26d5506990c228aff05ec9c88- Stable URL of this page
https://metraipolis.com/documento/dc_bfa605bd610d
How to cite
arXiv 2610.26645: Reasoning-Aware Multicalibrated Reliability Predictors for Language Models — Metraipolis archive, https://arxiv.org/abs/2610.26645 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_bfa605bd610d (Metraipolis archive dc_bfa605bd610d, sha256 b7563752b05283ade3b87879490c29a492b501b26d5506990c228aff05ec9c88)
BibTeX
@misc{dcbfa605bd610d,
title = {arXiv 2610.26645: Reasoning-Aware Multicalibrated Reliability Predictors for Language Models},
howpublished = {Metraipolis archive},
url = {https://arxiv.org/abs/2610.26645},
note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_bfa605bd610d; sha256 b7563752b05283ade3b87879490c29a492b501b26d5506990c228aff05ec9c88},
year = {2026}
}Archived text
Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://arxiv.org/abs/2610.26645. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
As people increasingly rely on large language models (LLMs) for information, model providers need to communicate meaningful estimates of whether individual responses are correct or, more generally, reliable. In the context of hallucinations, existing work uses multicalibration to make such reliability predictions meaningful across application-relevant groups, such as prompt topics. Topic groups alone, however, can conceal systematic errors: within the same topic and confidence level, responses with different chain-of-thought (CoT) properties may exhibit opposing errors that cancel on average. In this work, we introduce reasoning-aware multicalibration, a framework that supplements application-relevant topic groups with groups derived from observable properties of CoT traces. We highlight that, theoretically, groupings based on CoT are "internally important", i.e., groupings for which the multicalibration guarantee is not societally crucial as it is for high-risk topics, but whose inclusion in the multicalibration procedure could easily and significantly improve predictive accuracy of the final reliability predictor. Empirically, we apply our framework to hallucination prediction and unsafe-response detection across a variety of benchmarks, and find that adding CoT groups to topic-based calibration reduces Brier score in both tasks. Interestingly, we find that multicalibrating with respect to CoT groups alone (without any topic information) already improves calibration across topics, suggesting that the predictor's errors depend on how the model reasoned, rather than what it was asked about in the prompt. Articolo: «Reasoning-Aware Multicalibrated Reliability Predictors for Language Models» — Fedorova, Ekaterina; Morosini, Alessandro; Podimata, Chara (2026-10-19), arXiv:2610.26645, categoria cs.CY (informatica e società). Fonte: https://arxiv.org/abs/2610.26645 Licenza dell'articolo indicata su arXiv: CC-BY-4.0 — https://creativecommons.org/licenses/by/4.0. Qui solo titolo, autori e abstract (non modificato), con attribuzione; il testo integrale resta su arXiv. Consultato il 2026-10-20T15:27:59Z dalla pagina pubblica dell'articolo (robots.txt rispettato). Depositato dalla Gilda dei Bibliotecari. Abstract in lingua originale.