Metraipolis › archive › dc_db00351d745a
Effectiveness of Hidden-State Probes for Localizing Reasoning Failures — arXiv:2610.22046
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.22046
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T10:04:19.540Z
- SHA-256 of the archived text
16dfe8007c5264f380998a2219bf2149fd10f107d59e2b6b6fa9553534b53760- Stable URL of this page
https://metraipolis.com/documento/dc_db00351d745a
How to cite
Effectiveness of Hidden-State Probes for Localizing Reasoning Failures — arXiv:2610.22046 — arXiv (en), https://arxiv.org/abs/2610.22046 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_db00351d745a (Metraipolis archive dc_db00351d745a, sha256 16dfe8007c5264f380998a2219bf2149fd10f107d59e2b6b6fa9553534b53760)
BibTeX
@misc{dcdb00351d745a,
title = {Effectiveness of Hidden-State Probes for Localizing Reasoning Failures — arXiv:2610.22046},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.22046},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_db00351d745a; sha256 16dfe8007c5264f380998a2219bf2149fd10f107d59e2b6b6fa9553534b53760},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22046. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Verifying the final answer of a long reasoning trace may show that the trace is wrong but it does not distinguish three cases. The problem may lie beyond the model's capability, another attempt would succeed, or one step in an otherwise recoverable trace made the trace fail. Standard hidden-state probes are trained on final-answer correctness, a label that conflates these cases. Monte Carlo rollouts distinguish the cases and locate the failing step at additional inference cost. We test the hidden states of 4B and 8B reasoning models for the failure type and the failing step and compare each probe with the strongest baseline that uses no hidden states. On incorrect 8B traces, the confidence probe, a calibrated correctness probe at a midpoint cutoff, misses 81% of cases in which a single step makes the trace unrecoverable and 47% of cases in which another attempt would succeed. A probe with a different training design over the same states reverses the ordering of these two kinds of failure, so either ordering can be read from those states. At both scales, no single tested probe locates the failing step more accurately than a rule based only on trace length, and at 4B a probe trained to predict that step fails to outperform this rule in any of 10 runs on identical traces. The confidence probe therefore indicates that an answer is wrong, but its failure-type ordering depends on its training design. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.22046 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:44:47.770Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).