Metraipolis › archive › dc_b6881bb49f4c
Rating the Raters: Rasch Measurement Theory for LLM Evaluation — arXiv:2608.27463
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.27463
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T10:47:01.399Z
- SHA-256 of the archived text
5a684771ee2164925f6278d54856157b2f3e4830d2f928b0b560303eb3894431- Stable URL of this page
https://metraipolis.com/documento/dc_b6881bb49f4c
How to cite
Rating the Raters: Rasch Measurement Theory for LLM Evaluation — arXiv:2608.27463 — arXiv (en), https://arxiv.org/abs/2608.27463 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b6881bb49f4c (Metraipolis archive dc_b6881bb49f4c, sha256 5a684771ee2164925f6278d54856157b2f3e4830d2f928b0b560303eb3894431)
BibTeX
@misc{dcb6881bb49f4c,
title = {Rating the Raters: Rasch Measurement Theory for LLM Evaluation — arXiv:2608.27463},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.27463},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b6881bb49f4c; sha256 5a684771ee2164925f6278d54856157b2f3e4830d2f928b0b560303eb3894431},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27463. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLMs now sit on every side of evaluation: as examinees scored on benchmarks, judges of other models' outputs, and raters of human-generated content. Each paradigm can be viewed as a measurement problem, where a latent property of an object is probed with items from an instrument (e.g., benchmark) by judges or raters. Standard evaluation practices often neglect the contributions of each core component to the end result, limiting our understanding of what is being measured. Rasch measurement theory (RMT) is well-suited to this problem. RMT decomposes ordinal ratings into separable facets on a common scale. It further provides a battery of diagnostics that can identify miscalibrated measurements and rater biases. We present a case study of RMT applied to the LLM-as-rater paradigm using the Measuring Hate Speech corpus, whose construct was itself built under RMT. We fit a series of many-facet Rasch models to annotations from nine LLMs spanning families and capability levels. Our analyses show that LLMs systematically differ from human raters in severity, item-level calibration, question-order robustness, target-identity sensitivity, and rating scale use, all of which standard evaluation practice would largely obscure. Overall, we argue that RMT belongs in the toolkit for evaluating LLM-as-examinee, -judge, and -rater paradigms. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.27463 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T10:33:42.317Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).