Metraipolis › archive › dc_ada80bde6ff7
Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study — arXiv:2609.07395
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.07395
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T15:05:23.853Z
- SHA-256 of the archived text
9e4ea742e52f4745a8cba7c67c79bb1f58174cae5457d2b3bf7563e521803bb5- Stable URL of this page
https://metraipolis.com/documento/dc_ada80bde6ff7
How to cite
Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study — arXiv:2609.07395 — arXiv (en), https://arxiv.org/abs/2609.07395 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ada80bde6ff7 (Metraipolis archive dc_ada80bde6ff7, sha256 9e4ea742e52f4745a8cba7c67c79bb1f58174cae5457d2b3bf7563e521803bb5)
BibTeX
@misc{dcada80bde6ff7,
title = {Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study — arXiv:2609.07395},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.07395},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ada80bde6ff7; sha256 9e4ea742e52f4745a8cba7c67c79bb1f58174cae5457d2b3bf7563e521803bb5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07395. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are no longer deployed only for single-turn conversation but increasingly act as agents that plan, call tools, retrieve evidence, maintain memory, and interact over long horizons, often together with other agents through multi-turn conversations. Therefore, knowing when to trust the agentic system is a prerequisite for safe deployment. However, existing work on quantifying uncertainty for LLMs was built almost entirely for single-turn question answering. This paper argues that errors and uncertainty arise from multi-turn conversations, environments, and tools rather than from a single-turn question answering setting. It comes late, however, and is compounded in a single score that is too coarse to represent the unreliability. We organize the literature with a three-axis taxonomy, (1) what the uncertainty is, (2) how it is estimated, and (3) where uncertainty arises during an agent pipeline. We investigate step-level and trajectory-level calibration and show with a simple counterexample that the first does not imply the second. Experiments on real agent traces across four models and up to a 50-step budget show that the proposed metric and reporting protocol (Trajectory-Checkpoint Expected Calibration Error, TC-ECE) can be computed and that step errors are coupled along a trajectory. We find that confidence estimates from the agent's own responses do not consistently outperform a simple baseline. The experiments also show that averaging all trajectories together can hide overconfidence at later stages, which becomes visible when results are analyzed across different horizons. In simpler terms, this paper identifies where the uncertainty comes from in the agentic system pipeline, how to teach agents to know when they are wrong, and why one confidence number is not enough. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-09 in cs.IR). Categorie arXiv: Information Retrieval (cs.IR). Pagina del lavoro: https://arxiv.org/abs/2609.07395 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T14:49:33.955Z dalla pagina https://arxiv.org/catchup/cs.IR/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).