Metraipolis › archive › dc_de2b30da5004

When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation — arXiv:2608.14659

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.14659
License (as archived)
CC0-1.0
Archived
2026-10-18T17:05:55.778Z
SHA-256 of the archived text
2201d9fd7709892fc8280b43da97176bacbfe6f2c7cbc0df20f253e79ed858ba
Stable URL of this page
https://metraipolis.com/documento/dc_de2b30da5004

How to cite

When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation — arXiv:2608.14659 — arXiv (en), https://arxiv.org/abs/2608.14659 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_de2b30da5004 (Metraipolis archive dc_de2b30da5004, sha256 2201d9fd7709892fc8280b43da97176bacbfe6f2c7cbc0df20f253e79ed858ba)
BibTeX
@misc{dcde2b30da5004,
  title = {When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation — arXiv:2608.14659},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14659},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_de2b30da5004; sha256 2201d9fd7709892fc8280b43da97176bacbfe6f2c7cbc0df20f253e79ed858ba},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14659. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models for code generation often produce incorrect solutions without reliable indicators of failure. We study whether uncertainty estimation methods developed for natural language transfer to code generation, and whether such signals can improve code generation via selective self-correction. We evaluate five uncertainty methods: mean token entropy, verbalized confidence, $P(\text{True})$, entropy ensembles, and semantic entropy probes, across three small code LLMs on HumanEval and BigCodeBench. We find that multi-sample $P(\text{True})$ achieves the strongest correlation with correctness, while all the other methods, including semantic entropy probes, yield only weak correlation. We then use these uncertainty signals to drive three self-correction policies: adaptive decoding, uncertainty-based regeneration, and verification-based regeneration. Our results reveal a stronger negative finding than anticipated: uncertainty-based self-correction fails to reliably improve Pass@1, degrading accuracy in 5 of 6 configurations across both benchmarks ($-3$pp to $-10$pp), and adaptive decoding degrades accuracy in 4 of 6 configurations. Only verification-based self-correction reliably improves Pass@1, with gains of $+6$ to $+26$ percentage points on HumanEval and $+8$ to $+20$ percentage points on BigCodeBench, scaling inversely with baseline strength. These findings replicate consistently across both benchmarks and suggest that cheap uncertainty estimators are insufficient on their own to improve code correctness, and that their practical value lies in serving as gating signals for costlier execution-based correction loops rather than as standalone substitutes for verification.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.14659 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:13.719Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).