Metraipolis › archive › dc_f456c58505c7
BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments — arXiv:2610.11116
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11116
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:27:51.056Z
- SHA-256 of the archived text
01a901cb18ef31565a3edb2ea153a29fcb7c3c8b1fff78117fe236c0c2236d28- Stable URL of this page
https://metraipolis.com/documento/dc_f456c58505c7
How to cite
BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments — arXiv:2610.11116 — arXiv (en), https://arxiv.org/abs/2610.11116 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f456c58505c7 (Metraipolis archive dc_f456c58505c7, sha256 01a901cb18ef31565a3edb2ea153a29fcb7c3c8b1fff78117fe236c0c2236d28)
BibTeX
@misc{dcf456c58505c7,
title = {BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments — arXiv:2610.11116},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11116},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f456c58505c7; sha256 01a901cb18ef31565a3edb2ea153a29fcb7c3c8b1fff78117fe236c0c2236d28},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11116. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
AI evaluation pipelines often produce conflicting judgments rather than clean labels. In pairwise LLM evaluation, this conflict is especially visible: disagreement can arise from ambiguous items, underspecified rubrics, heterogeneous or unstable human raters, or an LLM judge whose verdict changes when the response order is swapped. We propose BayesJudge, an online Bayesian meta-evaluation layer for conflicting human-LLM judgment streams. For each comparison, BayesJudge estimates a panel-relative posterior verdict distribution over the two responses, with a tie or ambiguity state when such labels are available. At the same time, it estimates rater-specific human confusion matrices and LLM presentation-order bias. The method uses tie-open labels to keep ambiguity observable and paired order-swapped judge calls to separate response quality from presentation effects. We formulate the exact online posterior recursion and use a scalable Rao-Blackwellized assumed-density SMC approximation for streaming inference. Controlled synthetic experiments demonstrate recovery of prespecified evaluator parameters and illustrate two protocol-level identifiability mechanisms: tie-open labels expose ambiguity mass, and order-swapped paired judgments separate item preference from position bias. On a real-world SummEval dataset, BayesJudge successfully detects systematic presentation-order effects in LLM judge outputs, infers distinct expert and crowdworker behavior signatures without rater metadata, and produces posterior uncertainty estimates that correlate with human disagreement. Our code is available at this https URL . Tema degli esploratori: statistica e scienza dei dati (elenco arXiv stat.AP, nuovi lavori del giorno). Categorie arXiv: Applications (stat.AP). Pagina del lavoro: https://arxiv.org/abs/2610.11116 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:52:01.593Z dalla pagina https://arxiv.org/list/stat.AP/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).