Metraipolis › archive › dc_12e2cc484012

Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge — arXiv:2611.05662

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.05662
License (as archived): CC0-1.0
Archived: 2026-11-05T18:16:41.576Z
SHA-256 of the archived text: 3298c79b7c45a9214879d589c2a1be3fb53ea6f08611191f6ca6e95c309a8aa0
Stable URL of this page: https://metraipolis.com/documento/dc_12e2cc484012

How to cite

Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge — arXiv:2611.05662 — arXiv (en), https://arxiv.org/abs/2611.05662 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_12e2cc484012 (Metraipolis archive dc_12e2cc484012, sha256 3298c79b7c45a9214879d589c2a1be3fb53ea6f08611191f6ca6e95c309a8aa0)

BibTeX

@misc{dc12e2cc484012,
  title = {Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge — arXiv:2611.05662},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05662},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_12e2cc484012; sha256 3298c79b7c45a9214879d589c2a1be3fb53ea6f08611191f6ca6e95c309a8aa0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05662. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating open-ended agent tasks increasingly falls to an Agent-as-a-Judge: an evaluator handed tools and the agent's full trajectory so it can verify a result against the environment. The premise is that a judge that gathers its own evidence should be grounded by it. We show that this grounding does not immunize the judge against confirmation bias: a single false pre-screening note, attributed to an authority and carrying no evidential bearing on the correct verdict, overturns up to 40% of a tool-using judge's correct verdicts across four judges and five datasets, even though the judge collects decisive evidence. The shift is driven by specific properties of the cue. We treat this cue as a controlled diagnostic of confirmation bias rather than an adversarial attack. Our central contribution is to explain how confirmation bias changes the judge's behavior in a trajectory. We classify changes in judge behavior and attribute changes to one of three failure modes: over-trust (verify, conclude correctly, then defer), hallucination (fabricate support for the flip), and under-investigation (stop at the first corroborating signal). For the two Claude judges, the failure is predominantly over-trust (67-75% of harmful flips), while smaller judges have error modes spread across all three. Lightweight mitigations blunt but never remove the bias.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.05662 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:12:48.996Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).