Metraipolis › archive › dc_304255c7dc04

Commit-first LLM judging inherits the judge's own errors — arXiv:2609.00088

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.00088
License (as archived)
CC0-1.0
Archived
2026-10-17T06:32:47.227Z
SHA-256 of the archived text
dbc5952a684cc8975dab63993aaacbe815f3f17caa4a28bf7afcc073eb6856a2
Stable URL of this page
https://metraipolis.com/documento/dc_304255c7dc04

How to cite

Commit-first LLM judging inherits the judge's own errors — arXiv:2609.00088 — arXiv (en), https://arxiv.org/abs/2609.00088 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_304255c7dc04 (Metraipolis archive dc_304255c7dc04, sha256 dbc5952a684cc8975dab63993aaacbe815f3f17caa4a28bf7afcc073eb6856a2)
BibTeX
@misc{dc304255c7dc04,
  title = {Commit-first LLM judging inherits the judge's own errors — arXiv:2609.00088},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.00088},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_304255c7dc04; sha256 dbc5952a684cc8975dab63993aaacbe815f3f17caa4a28bf7afcc073eb6856a2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00088. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as ineffective, and share one ancestor prompt, traceable through a copied typographical error. In a controlled experiment, an ordinary best-of-N search with no access to correct answers optimises code against one of these configurations, used exactly as documented. On an interval merging task the judge accepted 90 of 96 candidates in one seed and 93 of 96 in the other; every accepted candidate passed every test the search could see and failed a held-out suite it could not. The judge identified the defective line and cited it as grounds for a perfect score. Commit-first judging removed the effect: 0 of 96 in both seeds. On a second task it made matters worse in both seeds: the judge's committed answer was wrong, and in one seed the population converged on it. This is our main finding. Commit-first judging does not remove the anchor that gets gamed, it moves it from the candidate to the judge's own answer, so evaluation is only as good as the judge is at the task. That precondition is cheap to measure in advance, and is task local rather than scale dependent: a smaller judge solved a task the frontier judge failed and resisted gaming where it did not. We also validate our own instruments: five of fifteen claims in our criteria were wrong against verbatim sources, and two held-out checks were unjustified by their specifications.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.00088 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:15:12.739Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).