Metraipolis › archive › dc_03365d9b9633

Human-Anchored Inference for Ranking New Models with Large Language Model Judges — arXiv:2609.19599

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.19599
License (as archived)
CC0-1.0
Archived
2026-10-10T04:21:03.148Z
SHA-256 of the archived text
0327ae5ba7834ec8ad27a876977d28f4aef19aed41ef24ce02b0847ee5bda9c7
Stable URL of this page
https://metraipolis.com/documento/dc_03365d9b9633

How to cite

Human-Anchored Inference for Ranking New Models with Large Language Model Judges — arXiv:2609.19599 — arXiv (en), https://arxiv.org/abs/2609.19599 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_03365d9b9633 (Metraipolis archive dc_03365d9b9633, sha256 0327ae5ba7834ec8ad27a876977d28f4aef19aed41ef24ce02b0847ee5bda9c7)
BibTeX
@misc{dc03365d9b9633,
  title = {Human-Anchored Inference for Ranking New Models with Large Language Model Judges — arXiv:2609.19599},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.19599},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_03365d9b9633; sha256 0327ae5ba7834ec8ad27a876977d28f4aef19aed41ef24ce02b0847ee5bda9c7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19599. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Human pairwise comparisons provide a reference for evaluating large language models (LLMs), but collecting sufficient judgments for each new release is costly and time-consuming. LLM judges offer a scalable alternative, although their comparisons may differ systematically from human preferences and across judges. We study the ranking of a new model that has received LLM-judge comparisons but no human comparisons. We propose ANCHOR (ANchored Comparisons for Human-reference inference with Orthogonal Riesz correction), which uses historical human and LLM comparisons to learn judge-specific sensitivities to human score differences and feature-dependent judge biases. These estimates are then used to infer the new model's human-reference score from its judge comparisons. The framework allows the feature distribution to change between historical and new-model comparisons. For inference, we construct a Neyman-orthogonal estimator through a joint Riesz correction that removes the first-order effects of estimating the historical human scores, judge sensitivities, and bias functions. We establish identification, convergence rates, and asymptotic normality with consistently estimable variance, and show that ANCHOR attains the semiparametric efficiency bound. Simulations demonstrate gains in score estimation and ranking accuracy. On Chatbot Arena, ANCHOR achieves the lowest score RMSE and insertion MAE among competing methods, with narrower score intervals on average.

Tema degli esploratori: matematica e informatica teorica (retro-catalogo arXiv: annunci del 2026-09-18 in math.ST).
Categorie arXiv: Statistics Theory (math.ST).
Pagina del lavoro: https://arxiv.org/abs/2609.19599 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T04:09:43.290Z dalla pagina https://arxiv.org/catchup/math.ST/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).