Metraipolis › archive › dc_5251c1fcae53
When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation — arXiv:2610.21128
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.21128
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T09:07:02.982Z
- SHA-256 of the archived text
3607b7db877a1f2f14079504a8538ec5861f94f41863ef42841c98799638408a- Stable URL of this page
https://metraipolis.com/documento/dc_5251c1fcae53
How to cite
When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation — arXiv:2610.21128 — arXiv (en), https://arxiv.org/abs/2610.21128 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5251c1fcae53 (Metraipolis archive dc_5251c1fcae53, sha256 3607b7db877a1f2f14079504a8538ec5861f94f41863ef42841c98799638408a)
BibTeX
@misc{dc5251c1fcae53,
title = {When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation — arXiv:2610.21128},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.21128},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5251c1fcae53; sha256 3607b7db877a1f2f14079504a8538ec5861f94f41863ef42841c98799638408a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21128. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Label-free evaluation methods, such as self-consistency filtering, majority-vote aggregation, and agreement-based reliability scoring, rely on an implicit assumption that consistency can serve as a proxy for correctness when ground-truth labels are unavailable or expensive to obtain. In this paper, we test this assumption directly. Using four financial QA datasets (multiple-choice and free-form), ten consistency dimensions, and two frontier model families, we measure the joint distribution of consistency and accuracy at the level of individual examples rather than in aggregate. We find that 60% of all incorrect answers fall in the high-consistency region (consistency $\geq$ 0.8), and that 77.7% of these show complete answer collapse, i.e. identical wrong answers across all perturbation variants. This high-consistency/incorrect quadrant cannot be detected by any evaluation signal that is built on agreement alone, as the model appears maximally reliable while being wrong. We show that the phenomenon is robust to the choice of consistency threshold t (it persists from t=0.50 to t=1.00), is shared across model families, and is concentrated in specific intersections of perturbation dimension and domain. We map where this quadrant occurs, analyze its stability properties, and discuss the implications for evaluation pipelines that use agreement-based signals in place of human labels. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.21128 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:44:17.909Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).