Metraipolis › archive › dc_9172ddcc3104
JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification — arXiv:2608.20607
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.20607
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T10:18:19.789Z
- SHA-256 of the archived text
525c91996b37dfec801b3a9357532a1e49a6819645a2a461237bddd5ac3f5e7f- Stable URL of this page
https://metraipolis.com/documento/dc_9172ddcc3104
How to cite
JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification — arXiv:2608.20607 — arXiv (en), https://arxiv.org/abs/2608.20607 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9172ddcc3104 (Metraipolis archive dc_9172ddcc3104, sha256 525c91996b37dfec801b3a9357532a1e49a6819645a2a461237bddd5ac3f5e7f)
BibTeX
@misc{dc9172ddcc3104,
title = {JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification — arXiv:2608.20607},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.20607},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9172ddcc3104; sha256 525c91996b37dfec801b3a9357532a1e49a6819645a2a461237bddd5ac3f5e7f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.20607. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Panels of inexpensive LLM judges increasingly make accept-or-escalate decisions. In factuality settings, accepting a claim because several reference-free judges agree can create a hidden risk: agreement may reflect shared false-negative blind spots rather than independent evidence. We introduce JuryProbe, an empirical consensus-risk diagnostic for reference-free factuality judge panels, paired with a calibration-based routing policy. JuryProbe estimates consensus risk from a labeled calibration probe using false-negative-only (FN-only) judge correlation and false-consensus lift; when flagged high-risk, reference-free majority accepts are routed to the same judges with trusted references. On audited FEVER corruptions, reference-free panels show correlated false negatives (FN-only correlations 0.402 and 0.368; lifts 3.13x and 18.13x), while unanimous false consensus drops to zero under a trusted-reference best-case diagnostic on both minimal-pair and non-minimal-pair evidence. In flagged settings, the routed policy is by construction equivalent to grounding every reference-free majority accept (verified in 34/34 splits): improvement comes from accept-conditioned grounding, while the diagnostic determines whether to activate it. A fixed, pre-specified rule flags 8-10 of 10 splits across synthetic, benchmark-authored, and scientific families and 0 of 10 on a negative control, where standing down avoids 28% of reference acquisitions at a 0.004 increase in false accepts. False-accept reduction persists under weak BM25 retrieval at substantial coverage cost, while stale stand-down labels require periodic recalibration. JuryProbe provides no formal risk guarantee and does not establish reliable stand-down on natural panels; its supported contribution is an empirical diagnostic of high-risk panel error dependence. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.20607 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T10:17:37.575Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).