Metraipolis › archive › dc_7e8ddd435ce9
Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces — arXiv:2609.38107
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38107
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:28:05.711Z
- SHA-256 of the archived text
99bcd94bbd636e3d12082f52be834595e38f53441f395fdeb6375e5d48b83cee- Stable URL of this page
https://metraipolis.com/documento/dc_7e8ddd435ce9
How to cite
Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces — arXiv:2609.38107 — arXiv (en), https://arxiv.org/abs/2609.38107 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7e8ddd435ce9 (Metraipolis archive dc_7e8ddd435ce9, sha256 99bcd94bbd636e3d12082f52be834595e38f53441f395fdeb6375e5d48b83cee)
BibTeX
@misc{dc7e8ddd435ce9,
title = {Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces — arXiv:2609.38107},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38107},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7e8ddd435ce9; sha256 99bcd94bbd636e3d12082f52be834595e38f53441f395fdeb6375e5d48b83cee},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38107. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Chain-of-thought traces are widely read as records of how models reach their answers, informing debugging, agent auditing, and claims about reasoning. Testing this interpretation is difficult because natural-language thinking traces are rarely mechanically verifiable. We revisit it in iGSM, a synthetic grade-school mathematics benchmark designed to study thinking traces and used to support claims of learned reasoning and planning. Crucially, iGSM exposes the exact quantities and dependencies that a correct solution should use, allowing generated traces to be checked programmatically step by step and enabling us to test whether correct answers are reliably accompanied by valid traces. We first evaluate models trained exclusively on valid, minimal traces. Answer correctness and trace validity nearly coincide in distribution but decouple out of distribution: on the hardest instances, 31.6% of correct answers have invalid traces, over half of which pass all syntactic and arithmetic checks but fail semantic dependency checks. We then intervene on trace supervision. Non-minimal training traces induce non-minimal outputs, while re-asking the same problem with a different query reveals computations inherited from the original query, weakening minimality as evidence of selective planning. Shuffling tokens in 10% of training trace sentences preserves near-clean accuracy even out of distribution despite no trace passing verification. Swapped training traces likewise retain high in-distribution accuracy. We discuss the implications of these findings for chain-of-thought monitoring and interpretation in the context of AI safety. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.38107 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T15:54:54.069Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).