Metraipolis › archive › dc_ae04c2eac8a4
AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory — arXiv:2611.05556
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.05556
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-05T18:16:37.753Z
- SHA-256 of the archived text
744b45f7ff763b2b7bcf9de803d79460b781de6defc52c07afd0fd5a818d1d33- Stable URL of this page
https://metraipolis.com/documento/dc_ae04c2eac8a4
How to cite
AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory — arXiv:2611.05556 — arXiv (en), https://arxiv.org/abs/2611.05556 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ae04c2eac8a4 (Metraipolis archive dc_ae04c2eac8a4, sha256 744b45f7ff763b2b7bcf9de803d79460b781de6defc52c07afd0fd5a818d1d33)
BibTeX
@misc{dcae04c2eac8a4,
title = {AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory — arXiv:2611.05556},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.05556},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ae04c2eac8a4; sha256 744b45f7ff763b2b7bcf9de803d79460b781de6defc52c07afd0fd5a818d1d33},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05556. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.05556 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-05T18:12:48.989Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).