Metraipolis › archive › dc_f0dbd761e755

Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation — arXiv:2611.01572

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01572
License (as archived)
CC0-1.0
Archived
2026-11-03T10:36:00.305Z
SHA-256 of the archived text
48b1ae7f5f5a132c34ef4406ab5286d45893977445f8377b0e880eba96c9e7fb
Stable URL of this page
https://metraipolis.com/documento/dc_f0dbd761e755

How to cite

Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation — arXiv:2611.01572 — arXiv (en), https://arxiv.org/abs/2611.01572 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f0dbd761e755 (Metraipolis archive dc_f0dbd761e755, sha256 48b1ae7f5f5a132c34ef4406ab5286d45893977445f8377b0e880eba96c9e7fb)
BibTeX
@misc{dcf0dbd761e755,
  title = {Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation — arXiv:2611.01572},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01572},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f0dbd761e755; sha256 48b1ae7f5f5a132c34ef4406ab5286d45893977445f8377b0e880eba96c9e7fb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01572. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models are often used to judge generated code without running it. A useful evaluation should reveal substantial changes in the answers it scores. We examine this requirement by repeating an earlier evaluation on 421 coding problems, comparing Qwen-2.5-7B-Instruct with two versions trained using Group Relative Policy Optimization (GRPO). One receives rewards from execution tests (CE-trained), and the other from a strong language-model judge (SLJ-trained). The base model's syntax-failure rate remains stable, but the trained checkpoints produce invalid Python on 63%-67% of attempts. Despite this change, the strong judge's Pass@32 scores fall by only about 3 percentage points, from 96.67% to 93.95% for SLJ-trained. Within the later run, allowing more attempts makes the base and trained checkpoints look increasingly similar under the judge. With 128 attempts per question, the judge accepts at least one SLJ-trained answer on 96.44% of problems, compared with 28.27% under execution tests. We recommend reporting the share of answers that pass Python's syntax check alongside every judge-based score. This check requires no tests or code execution and flags invalid answers that the judge accepts, although it cannot establish functional correctness. Our findings show how a high judge-based score can hide substantial output damage and motivate checking an evaluation's sensitivity to change. Project page and code: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01572 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:29:48.786Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).