Metraipolis › archive › dc_c5803bf397de

Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation — arXiv:2611.01572

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01572
License (as archived)
CC0-1.0
Archived
2026-11-03T10:56:46.310Z
SHA-256 of the archived text
3dacbbaa12b19bce2c53b802fb933cb139997f794d2662e83de8296c61597f74
Stable URL of this page
https://metraipolis.com/documento/dc_c5803bf397de

How to cite

Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation — arXiv:2611.01572 — arXiv (en), https://arxiv.org/abs/2611.01572 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c5803bf397de (Metraipolis archive dc_c5803bf397de, sha256 3dacbbaa12b19bce2c53b802fb933cb139997f794d2662e83de8296c61597f74)
BibTeX
@misc{dcc5803bf397de,
  title = {Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation — arXiv:2611.01572},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01572},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c5803bf397de; sha256 3dacbbaa12b19bce2c53b802fb933cb139997f794d2662e83de8296c61597f74},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01572. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models are often used to judge generated code without running it. A useful evaluation should reveal substantial changes in the answers it scores. We examine this requirement by repeating an earlier evaluation on 421 coding problems, comparing Qwen-2.5-7B-Instruct with two versions trained using Group Relative Policy Optimization (GRPO). One receives rewards from execution tests (CE-trained), and the other from a strong language-model judge (SLJ-trained). The base model's syntax-failure rate remains stable, but the trained checkpoints produce invalid Python on 63%-67% of attempts. Despite this change, the strong judge's Pass@32 scores fall by only about 3 percentage points, from 96.67% to 93.95% for SLJ-trained. Within the later run, allowing more attempts makes the base and trained checkpoints look increasingly similar under the judge. With 128 attempts per question, the judge accepts at least one SLJ-trained answer on 96.44% of problems, compared with 28.27% under execution tests. We recommend reporting the share of answers that pass Python's syntax check alongside every judge-based score. This check requires no tests or code execution and flags invalid answers that the judge accepts, although it cannot establish functional correctness. Our findings show how a high judge-based score can hide substantial output damage and motivate checking an evaluation's sensitivity to change. Project page and code: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01572 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.569Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).