Metraipolis › archive › dc_ec84cdfdd829
Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses — arXiv:2610.30200
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.30200
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-25T20:57:37.312Z
- SHA-256 of the archived text
9fe53fa8d71a2a2e0a58fc88f8b24fc91842041f14aa21d855d6e5127183e39a- Stable URL of this page
https://metraipolis.com/documento/dc_ec84cdfdd829
How to cite
Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses — arXiv:2610.30200 — arXiv (en), https://arxiv.org/abs/2610.30200 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ec84cdfdd829 (Metraipolis archive dc_ec84cdfdd829, sha256 9fe53fa8d71a2a2e0a58fc88f8b24fc91842041f14aa21d855d6e5127183e39a)
BibTeX
@misc{dcec84cdfdd829,
title = {Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses — arXiv:2610.30200},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.30200},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ec84cdfdd829; sha256 9fe53fa8d71a2a2e0a58fc88f8b24fc91842041f14aa21d855d6e5127183e39a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30200. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
When a language model is deployed, no reference solution exists, so the reasoning trace it writes is the only evidence for judging its answer. The reference-free substitutes each fail differently: self-assessment degrades the answer it was meant to protect, sampling agreement measures stability rather than correctness, and perturbation studies read the expected answer off a human-written template, grading a model rather than a trace. We ask how far a single trace can be verified with no reference solution, no judge model and no weight update, and find, for a frozen LLaMA-2-7B-chat on GSM8K, that detection is not what limits such a pipeline. Checking a trace against itself reaches an AUROC of 0.741 and stops there for a structural reason: two thirds of wrong traces state no false equation, having executed a mistaken plan correctly, and on wrong-plan failures the detector flags a wrong trace less often than a right one. A candidate-derived metamorphic probe exposes many of these failures: we abstract the trace into a program over the question's quantities, perturb one quantity, execute the program, and ask the model the rewritten question once, so every trace supplies its own test. Valid perturbations exist for 937 of 1,319 test questions (71.0%); among the 847 that also have a parseable first trace, supported traces are correct 60.6% of the time against a 31.6% base rate, although the probe also refutes 45.5% of correct traces. No tested policy yields a statistically significant improvement in accuracy. A flag earns its retry only when detector precision exceeds D/(R+D), where R and D are the rates at which a retry repairs a wrong trace and damages a correct one; our repair rate is so low that, at the same retry budget, even a perfect detector would recover an estimated 66 of 1,319 questions. Repair, not detection, is the open problem. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.30200 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-25T20:47:56.965Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).