Metraipolis › archive › dc_e2a466fce980
VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation — arXiv:2611.00491
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00491
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:19:24.666Z
- SHA-256 of the archived text
138932cce19d39167d066e2e98c7252e324dfb63a24c880245059d56b645b0f7- Stable URL of this page
https://metraipolis.com/documento/dc_e2a466fce980
How to cite
VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation — arXiv:2611.00491 — arXiv (en), https://arxiv.org/abs/2611.00491 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e2a466fce980 (Metraipolis archive dc_e2a466fce980, sha256 138932cce19d39167d066e2e98c7252e324dfb63a24c880245059d56b645b0f7)
BibTeX
@misc{dce2a466fce980,
title = {VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation — arXiv:2611.00491},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00491},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e2a466fce980; sha256 138932cce19d39167d066e2e98c7252e324dfb63a24c880245059d56b645b0f7},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00491. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM-as-a-judge systems often aggregate multiple judges to improve evaluation reliability. However, judge agreement can reflect shared biases toward verbosity, formatting, or fluency rather than the intended evaluation target. Addressing these shared biases is challenging because observational judge outputs alone cannot distinguish agreement driven by target quality from agreement driven by nuisance attributes. We introduce VoirDire, an intervention-calibrated aggregation method for LLM-judge panels. VoirDire applies controlled interventions that vary specified surface attributes while aiming to preserve the evaluation target; it then uses paired changes in judge scores to identify nuisance-sensitive directions. It attenuates these directions and selects aggregation weights that retain variation across items while penalizing sensitivity to the interventions, making the final consensus less dependent on superficial features. We show theoretically that observational judge outputs alone cannot identify which shared direction is target-relevant, while paired interventions provide the contrast needed to identify nuisance-sensitive variation. We evaluate VoirDire on 12 datasets spanning continuous scoring, binary classification, and pairwise preference. Compared with the existing LLM-judge aggregation baselines, VoirDire reduces MAE by up to 26.8% on continuous-score tasks and improves accuracy by up to 5.3 percentage points on binary and preference tasks. These findings suggest that counterfactual interventions provide an additional means of improving the reliability and controllability of LLM-based evaluation beyond merely relying on observed judge agreement. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.00491 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:02:37.444Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).