Metraipolis › archive › dc_f1dee4388ee9
LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard — arXiv:2610.32986
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.32986
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T08:12:32.182Z
- SHA-256 of the archived text
189d180032bfc86b9a7f6dfebf309bc2500a9ad1104e9e7a94ffa47455f60e14- Stable URL of this page
https://metraipolis.com/documento/dc_f1dee4388ee9
How to cite
LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard — arXiv:2610.32986 — arXiv (en), https://arxiv.org/abs/2610.32986 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f1dee4388ee9 (Metraipolis archive dc_f1dee4388ee9, sha256 189d180032bfc86b9a7f6dfebf309bc2500a9ad1104e9e7a94ffa47455f60e14)
BibTeX
@misc{dcf1dee4388ee9,
title = {LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard — arXiv:2610.32986},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.32986},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f1dee4388ee9; sha256 189d180032bfc86b9a7f6dfebf309bc2500a9ad1104e9e7a94ffa47455f60e14},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32986. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Automated research pipelines increasingly rely on fixed LLM judges to rank, filter, and revise proposals, yet these judges are routinely replaced by newer versions. We study whether such replacements preserve the underlying measurement standard. Using a preregistered, frozen-weight design, we evaluate three adjacent same-family transitions across Qwen, Gemma, and Mistral on frozen ML research proposals under randomized information conditions. The common assumption that newer judges are simply harsher fails: two transitions shift toward greater leniency, while the third shifts in the opposite direction on anchors but attenuates off-anchor. More importantly, version changes can alter not only score location but also criterion geometry and resolution; one transition collapses clean-flawed discrimination, placing 93% of pool scores on a single value. Supplying identical facts does not close the version gap, placebo-controlled interventions show that observed packet effects are largely format-driven rather than content-driven, and global recalibration repairs the mean while explaining little object-level dispersion. We organize these failures with a measurement model in which a judge replacement can change its origin, criterion geometry, or resolution. A small stratified anchor audit of about 200 calls identifies the failure class before deployment, and frozen thresholds correctly diagnose two post-registration API judge replacements. We release six preregistration locks and the full audit trail of 14882 judge calls. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computers and Society (cs.CY) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.32986 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-27T07:56:00.282Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).