Metraipolis › archive › dc_720443eec151

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty — arXiv:2610.02877

Source: arXiv
Original: https://arxiv.org/abs/2610.02877
License (as archived): CC0-1.0
Archived: 2026-10-05T22:51:37.179Z
SHA-256 of the archived text: e0ccc753660774017f19d0b61442b9e51ad56c8ebce59971bec5316db38facdf
Stable URL of this page: https://metraipolis.com/documento/dc_720443eec151

How to cite

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty — arXiv:2610.02877 — arXiv, https://arxiv.org/abs/2610.02877 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_720443eec151 (Metraipolis archive dc_720443eec151, sha256 e0ccc753660774017f19d0b61442b9e51ad56c8ebce59971bec5316db38facdf)

BibTeX

@misc{dc720443eec151,
  title = {Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty — arXiv:2610.02877},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02877},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_720443eec151; sha256 e0ccc753660774017f19d0b61442b9e51ad56c8ebce59971bec5316db38facdf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02877. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases difficult. In this paper, we study this problem in summarization evaluation from a psychometric perspective. We fit Many-Facet Rasch Models separately to human and LLM ratings to decompose scores into latent summary quality, rater severity, dimension severity, and rating-scale thresholds. Building on this decomposition, we define residual hardness as a model-adjusted measure of judging difficulty and compare whether human and LLM judges share the same hardness structure. Across 17 open-weight LLM judges on SummEval, we find that moderate alignment in latent summary quality does not imply alignment in residual hardness. Human and LLM judges differ in which summary--dimension units remain difficult, and this mismatch is strongly dimension-dependent. Consistency shows a pronounced LLM-hard shift, whereas coherence shows a human-hard shift. We further show that human-easy but LLM-hard cases are partially predictable from observable source--summary properties. These findings suggest that aggregate human alignment reflects only part of LLM-as-a-judge reliability, while psychometric residual diagnostics support more informative judge evaluation and more targeted human--LLM collaboration.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.02877 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:46:10.556Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).