Metraipolis › archive › dc_9530ee7548ca
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation — arXiv:2609.12439
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.12439
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:03:23.092Z
- SHA-256 of the archived text
bdc7347aef82aa17ffd94ae6eb34e6b655812498e23dce483bbdc35e16978f1d- Stable URL of this page
https://metraipolis.com/documento/dc_9530ee7548ca
How to cite
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation — arXiv:2609.12439 — arXiv (en), https://arxiv.org/abs/2609.12439 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9530ee7548ca (Metraipolis archive dc_9530ee7548ca, sha256 bdc7347aef82aa17ffd94ae6eb34e6b655812498e23dce483bbdc35e16978f1d)
BibTeX
@misc{dc9530ee7548ca,
title = {Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation — arXiv:2609.12439},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.12439},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9530ee7548ca; sha256 bdc7347aef82aa17ffd94ae6eb34e6b655812498e23dce483bbdc35e16978f1d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12439. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM-as-a-judge protocols are commonly debiased by instructing judges to ignore presentation cues such as citation formatting, source labels, and evidence-display style. We show that this intervention can suppress bias while damaging the resolution of the measurement instrument. We introduce TraceJudgeBench, a diagnostic benchmark for auditing citation-like artifacts in RAG and agent-workflow evaluation, covering content-equivalent pairs, citation ablations, correctness conflicts, human-validated soft and moderate quality gaps, prompt-strength ladders, decoupled judging, and a controlled workflow-ranking probe. Across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash, stronger anti-citation prompts reduce worse-cited wins from up to 50.5% to 0%; yet some operating points already convert validated moderate-gap decisions into Tie before the strict stress-test endpoint, while correctness-conflict accuracy remains at or above 93.0%. A second, 50-pair FinQA moderate-gap construction reproduces the qualitative frontier, and open-weight Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT runs reproduce the central HotpotQA frontier. TRACE-style decoupling recovers 96.5-100.0% better-plain resolution across the reported settings. Human validation separates three meanings of Tie: correct equivalence Tie, calibrated soft-boundary Tie, and resolution-destroying Tie on validated quality gaps. We frame debiasing as a measurement intervention whose bias suppression, resolution retention, Tie cost, and protocol cost must be reported jointly. The supplementary artifact contains benchmark splits, prompts, raw judge outputs, validation summaries, and analysis. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.AI). Categorie arXiv: Digital Libraries (cs.DL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.12439 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T02:51:27.906Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).