Metraipolis › archive › dc_7d34397c87ab
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation — arXiv:2608.30373
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.30373
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T08:07:29.923Z
- SHA-256 of the archived text
4b98e152ef3487f353a44cdc8778f298c3faf6823c049663cb47b0c888ac4d71- Stable URL of this page
https://metraipolis.com/documento/dc_7d34397c87ab
How to cite
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation — arXiv:2608.30373 — arXiv (en), https://arxiv.org/abs/2608.30373 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d34397c87ab (Metraipolis archive dc_7d34397c87ab, sha256 4b98e152ef3487f353a44cdc8778f298c3faf6823c049663cb47b0c888ac4d71)
BibTeX
@misc{dc7d34397c87ab,
title = {Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation — arXiv:2608.30373},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.30373},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d34397c87ab; sha256 4b98e152ef3487f353a44cdc8778f298c3faf6823c049663cb47b0c888ac4d71},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30373. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human alignment on average across six judge models, whereas MAD shows degradation in human alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-agent disagreement on average and worsens average human alignment. These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.30373 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:47:52.804Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).