Metraipolis › archive › dc_2c134ef67636
JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places — arXiv:2609.29769
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.29769
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T04:49:44.238Z
- SHA-256 of the archived text
c1cf2cf5cf036223e49598407c29f92e6977ddca67f0541c3c9ffc9a9a82ed7f- Stable URL of this page
https://metraipolis.com/documento/dc_2c134ef67636
How to cite
JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places — arXiv:2609.29769 — arXiv (en), https://arxiv.org/abs/2609.29769 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2c134ef67636 (Metraipolis archive dc_2c134ef67636, sha256 c1cf2cf5cf036223e49598407c29f92e6977ddca67f0541c3c9ffc9a9a82ed7f)
BibTeX
@misc{dc2c134ef67636,
title = {JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places — arXiv:2609.29769},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.29769},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2c134ef67636; sha256 c1cf2cf5cf036223e49598407c29f92e6977ddca67f0541c3c9ffc9a9a82ed7f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29769. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM judges score outputs against rubrics well enough to have become the norm, both in benchmarks and as rewards for training. Jev, a classifier-like alternative its creators call a "decision model", returns probabilities over permitted answers with a calibrated confidence score, which LLM judges do not natively provide. We compare Jev with three flash-tier LLM judges on nine panels from seven benchmarks with human judgments, giving every judge identical criterion texts. The LLM judges run in two setups: holistically, reading a whole rubric at once as Jev does, and one criterion at a time. Jev can often stand in for them. They cost 16 to 325 times as much and take 28 to 350 times as long, yet in each setup Jev's accuracy differs significantly from theirs in at most 8 of 27 paired comparisons, ahead mostly on binary checklist criteria and behind only on ordinal ones. Despite their different designs, the two kinds of judge err alike. On ordinal criteria, all LLM judges and Jev depart from the human raters together, agreeing more with one another than with the labels and mostly assigning lower levels. On Jev's most confident errors, about 96% of LLM verdicts repeat its wrong answer, where independent errors would give about half. Intuitively, calibrated confidence should make Jev an ideal first stage of a cascade that defers uncertain verdicts to an LLM judge. Yet such cascades only lower cost while adding little accuracy: even with oracle thresholds, none beats the best single judge by more than 2.7 points. Calibration can tell a cascade when to defer, but the cascade also needs a fallback that errs elsewhere; these judges are wrong in the same places. These findings, which hold in both setups and at high reasoning effort, suggest that a cascade of judges succeeds only when its judges make complementary errors, and that future decision models should be designed afresh with that aim. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.29769 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T04:27:20.098Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).