Metraipolis › archive › dc_649ee0be3f8d

Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels — arXiv:2609.17857

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.17857
License (as archived)
CC0-1.0
Archived
2026-10-15T13:49:17.107Z
SHA-256 of the archived text
db66129a0f2a04a20cbd41e5b0e4f3d77ed15b10c1c03eaef0a34905712cabf9
Stable URL of this page
https://metraipolis.com/documento/dc_649ee0be3f8d

How to cite

Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels — arXiv:2609.17857 — arXiv (en), https://arxiv.org/abs/2609.17857 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_649ee0be3f8d (Metraipolis archive dc_649ee0be3f8d, sha256 db66129a0f2a04a20cbd41e5b0e4f3d77ed15b10c1c03eaef0a34905712cabf9)
BibTeX
@misc{dc649ee0be3f8d,
  title = {Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels — arXiv:2609.17857},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.17857},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_649ee0be3f8d; sha256 db66129a0f2a04a20cbd41e5b0e4f3d77ed15b10c1c03eaef0a34905712cabf9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.17857. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwise design with 9,312 judgments. A common per-family statistic is strongly confounded with candidate quality and correlates with Bradley-Terry ability at r = 0.95. We derive a corrected estimator that holds the candidate family fixed and compares judges. All four families then show a positive same-family lift (3.4-8.4 percentage points), with global FPS 0.067 (95% CI [0.053, 0.084], permutation p = 0.0002). The effect remains under panel-based quality controls, an independent human-consensus anchor, and a float16 judging replication. Judge-side likelihood is closely related to the effect: adding likelihood advantage reduces the controlled coefficient by 61%, which we treat as descriptive attenuation rather than causal mediation. Position is a separate failure mode. Across the panel, 55.4% of AB/BA pairs reverse, and reversal above 50% is incompatible with a simple independent content-noise model. Relative to a family-balanced reference, panel composition changes 18.5% of pairwise outcomes. A complete reproducibility archive has been prepared for public release.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.17857 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:48:24.284Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).