Metraipolis › archive › dc_0ccc1db0be5d

Which Signals of Code Correctness Survive Distribution Shift? Hidden-State and Causal Predictors of Program and Verdict Correctness — arXiv:2611.07876

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.07876
License (as archived)
CC0-1.0
Archived
2026-11-06T14:51:16.045Z
SHA-256 of the archived text
f9454b602638b90ce8b331fab06884c9b0f021415ab87fde5441959ff1e0ab72
Stable URL of this page
https://metraipolis.com/documento/dc_0ccc1db0be5d

How to cite

Which Signals of Code Correctness Survive Distribution Shift? Hidden-State and Causal Predictors of Program and Verdict Correctness — arXiv:2611.07876 — arXiv (en), https://arxiv.org/abs/2611.07876 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ccc1db0be5d (Metraipolis archive dc_0ccc1db0be5d, sha256 f9454b602638b90ce8b331fab06884c9b0f021415ab87fde5441959ff1e0ab72)
BibTeX
@misc{dc0ccc1db0be5d,
  title = {Which Signals of Code Correctness Survive Distribution Shift? Hidden-State and Causal Predictors of Program and Verdict Correctness — arXiv:2611.07876},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07876},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ccc1db0be5d; sha256 f9454b602638b90ce8b331fab06884c9b0f021415ab87fde5441959ff1e0ab72},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07876. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models increasingly judge code, yet without external feedback such judgments are unreliable, and verifying them by executing tests or by extended reasoning is costly. Instead, execution-free correctness predictors use signals from the model's output probabilities, sampled solutions, or hidden states to identify judgments that warrant further verification. However, these predictors may degrade under distribution shifts when generalizing across projects and languages. We compare six execution-free predictors from four families on Qwen3-Coder-30B and gpt-oss-20b under benchmark shift between LiveCodeBench and BigCodeBench and language shift across Python, C++ and Java. We evaluate two targets: whether the model's own verdict on a program can be trusted, and whether the program itself is correct. Besides established approaches, we adapt two novel causal localization methods to code, which have been shown to be more robust out-of-distribution on natural language tasks. For verdict correctness, causal localization improves robustness to language shift. For Qwen3-Coder a probe on the causally localized subspace stays 0.09 AUROC ahead of output confidence under language shift (0.11 in distribution), significantly outperforming the best non-causal predictor by 0.04 AUROC on average. This advantage, however, does not generalize consistently to gpt-oss or to benchmark shift. For program correctness, predictors transfer more successfully between benchmarks, largely by identifying problems on which the model tends to fail. However, their ability to rank candidates within the same problem is limited.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.07876 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:21:42.665Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).