Metraipolis › archive › dc_84fc08d8a555
Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction — arXiv:2609.19606
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.19606
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T03:48:02.184Z
- SHA-256 of the archived text
3e3b56fedecbd09f0003ad48a5802b76e7aa36f3fd66e56713906039efffd0db- Stable URL of this page
https://metraipolis.com/documento/dc_84fc08d8a555
How to cite
Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction — arXiv:2609.19606 — arXiv (en), https://arxiv.org/abs/2609.19606 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84fc08d8a555 (Metraipolis archive dc_84fc08d8a555, sha256 3e3b56fedecbd09f0003ad48a5802b76e7aa36f3fd66e56713906039efffd0db)
BibTeX
@misc{dc84fc08d8a555,
title = {Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction — arXiv:2609.19606},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.19606},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84fc08d8a555; sha256 3e3b56fedecbd09f0003ad48a5802b76e7aa36f3fd66e56713906039efffd0db},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19606. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
This empirical study is an independent reproduction of the dissociation Zhao reported in 2026. The shape of a large language model's chain-of-thought entropy trajectory predicts whether the final answer is correct, while the magnitude of its total entropy drop does not. The dissociation merits reproduction because the magnitude half rests on a single 300-problem run with one model at one seed, while the shape half was reported at full scale on both benchmarks and on a second model family. Registered at OSF before any confirmatory run, the reproduction crosses the complete GSM8K and MATH-500 benchmark test sets with four open-weight models including one reasoning-distilled model of a kind the original did not test. The shape signal replicates. The magnitude signal divides by setting. On the anchor model the accuracy gap between monotone and non-monotone chains is +9.6 percentage points on GSM8K and +27.5 on MATH-500, while the rank correlation of the total entropy drop with correctness is -0.018 on GSM8K and +0.414 on MATH-500. On the reasoning-distilled model the binary form of the shape signal fires on about one chain in a hundred, too few to estimate the registered contrast, while the graded violation count remains predictive there. In an exploratory comparison the final-step entropy alone outperforms the binary shape flag in all eight model-by-benchmark cells by ROC area, and in six or seven by the risk-coverage area the original reports, depending on an integration range the original does not state. The study contributes a reproduction of the shape signal at full test-set scale under seven documented protocol differences, a map of the settings where the magnitude signal holds and fails, and measurements of four protocol dependencies the original does not report. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.19606 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-10T03:24:08.127Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).