Metraipolis › archive › dc_84fc08d8a555

Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction — arXiv:2609.19606

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.19606
License (as archived)
CC0-1.0
Archived
2026-10-10T03:48:02.184Z
SHA-256 of the archived text
3e3b56fedecbd09f0003ad48a5802b76e7aa36f3fd66e56713906039efffd0db
Stable URL of this page
https://metraipolis.com/documento/dc_84fc08d8a555

How to cite

Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction — arXiv:2609.19606 — arXiv (en), https://arxiv.org/abs/2609.19606 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84fc08d8a555 (Metraipolis archive dc_84fc08d8a555, sha256 3e3b56fedecbd09f0003ad48a5802b76e7aa36f3fd66e56713906039efffd0db)
BibTeX
@misc{dc84fc08d8a555,
  title = {Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction — arXiv:2609.19606},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.19606},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84fc08d8a555; sha256 3e3b56fedecbd09f0003ad48a5802b76e7aa36f3fd66e56713906039efffd0db},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19606. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This empirical study is an independent reproduction of the dissociation Zhao reported in 2026. The shape of a large language model's chain-of-thought entropy trajectory predicts whether the final answer is correct, while the magnitude of its total entropy drop does not. The dissociation merits reproduction because the magnitude half rests on a single 300-problem run with one model at one seed, while the shape half was reported at full scale on both benchmarks and on a second model family. Registered at OSF before any confirmatory run, the reproduction crosses the complete GSM8K and MATH-500 benchmark test sets with four open-weight models including one reasoning-distilled model of a kind the original did not test. The shape signal replicates. The magnitude signal divides by setting. On the anchor model the accuracy gap between monotone and non-monotone chains is +9.6 percentage points on GSM8K and +27.5 on MATH-500, while the rank correlation of the total entropy drop with correctness is -0.018 on GSM8K and +0.414 on MATH-500. On the reasoning-distilled model the binary form of the shape signal fires on about one chain in a hundred, too few to estimate the registered contrast, while the graded violation count remains predictive there. In an exploratory comparison the final-step entropy alone outperforms the binary shape flag in all eight model-by-benchmark cells by ROC area, and in six or seven by the risk-coverage area the original reports, depending on an integration range the original does not state. The study contributes a reproduction of the shape signal at full test-set scale under seven documented protocol differences, a map of the settings where the magnitude signal holds and fails, and measurements of four protocol dependencies the original does not report.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.19606 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:24:08.127Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).