Metraipolis › archive › dc_5b1dfee31c4f
Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay — arXiv:2608.26846
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.26846
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T14:30:10.618Z
- SHA-256 of the archived text
0e1144384192c2a79bec3e526391466bfd9fa6b8b100902a299c41ee20fa1129- Stable URL of this page
https://metraipolis.com/documento/dc_5b1dfee31c4f
How to cite
Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay — arXiv:2608.26846 — arXiv (en), https://arxiv.org/abs/2608.26846 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5b1dfee31c4f (Metraipolis archive dc_5b1dfee31c4f, sha256 0e1144384192c2a79bec3e526391466bfd9fa6b8b100902a299c41ee20fa1129)
BibTeX
@misc{dc5b1dfee31c4f,
title = {Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay — arXiv:2608.26846},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.26846},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5b1dfee31c4f; sha256 0e1144384192c2a79bec3e526391466bfd9fa6b8b100902a299c41ee20fa1129},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26846. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Interactive language-model agents use confidence signals to decide whether to answer immediately, retrieve additional evidence (from memory or external knowledge), or defer. Yet confidence is usually evaluated in isolation, without measuring the trajectory-level consequences of the actions it triggers. We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings. The protocol holds candidate answer states, evidence points, budgets, and action costs fixed. We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system using Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. At the same numerical commitment threshold, calibration changes which questions agents ultimately commit to answering. Across all six model-dataset pairs, it increases accuracy among committed answers by up to 41 percentage points. However, it can reduce coverage and increase retrieval use. Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue. These effects reflect a shift to a more selective, lower-risk operating point, not improved answers or confidence ranking. A calibration map fitted before retrieval improves held-out calibration through retrieval depths one and two, but is worse than raw confidence at depth three for all three models. Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval. Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval. Retrieval therefore requires a separate value-of-information or utility estimate. Evaluations should report held-out calibration, risk-coverage, and retrieval cost. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.26846 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T13:16:53.666Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).