Metraipolis › archive › dc_b562c869b762

Confident at the moment of action: belief miscalibration in LLM play under hidden information — arXiv:2608.24691

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.24691
License (as archived)
CC0-1.0
Archived
2026-10-18T06:20:55.888Z
SHA-256 of the archived text
31514529dd9ff66e39c8f3c759b3c76b97d57412bc4e08fcfa41ab6a8065c3c0
Stable URL of this page
https://metraipolis.com/documento/dc_b562c869b762

How to cite

Confident at the moment of action: belief miscalibration in LLM play under hidden information — arXiv:2608.24691 — arXiv (en), https://arxiv.org/abs/2608.24691 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b562c869b762 (Metraipolis archive dc_b562c869b762, sha256 31514529dd9ff66e39c8f3c759b3c76b97d57412bc4e08fcfa41ab6a8065c3c0)
BibTeX
@misc{dcb562c869b762,
  title = {Confident at the moment of action: belief miscalibration in LLM play under hidden information — arXiv:2608.24691},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.24691},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b562c869b762; sha256 31514529dd9ff66e39c8f3c759b3c76b97d57412bc4e08fcfa41ab6a8065c3c0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24691. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agentic systems increasingly gate actions on a model's own stated confidence, which assumes confidence tracks correctness at the moment of acting. We test this in a hidden-information chess variant where royal status can be secretly, repeatedly relocated between pieces, and where an agent's stated probability distribution over the opponent's hidden royal piece -- elicited every turn, separately from the move it chooses -- is scored against ground truth recoverable after the game. Across two independent batches, captures made at high stated confidence ($\geq 0.5$) about the hidden piece's location were correct in 1 of 62 cases. The calibration deficit is concentrated almost entirely in these events: 99.3% of it in the original batch, 98.7% in the replication. The same pattern, in weaker form, orders consistently (point estimates only; most pairwise gaps are not statistically distinguishable at this sample size) across four further model configurations spanning a second provider -- reported as scope for the finding, not as evidence that capability predicts calibration: a same-model comparison at a fixed external leaderboard score shows a deliberation-budget change alone moves the metric by nearly as much as a large cross-model gap. In a separate seat, conventional evaluation axes -- legality, cost, latency, completion rate -- can dissociate entirely from belief quality, with the configuration winning on every conventional axis producing the worst belief quality tested. A model exhibiting this pattern can still win the game its belief was about, which is why outcome-only evaluation would not detect it.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.CL).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.24691 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T06:17:29.417Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).