Metraipolis › archive › dc_c4beee58e5b8

JudgeProfile: Understanding and Steering Subjectivity in LLM Judges — arXiv:2609.36705

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36705
License (as archived)
CC0-1.0
Archived
2026-10-07T16:22:54.147Z
SHA-256 of the archived text
a9593aa5094548df8e4d844c352ee7a579b20af4273cf768b08f0504edc49a2c
Stable URL of this page
https://metraipolis.com/documento/dc_c4beee58e5b8

How to cite

JudgeProfile: Understanding and Steering Subjectivity in LLM Judges — arXiv:2609.36705 — arXiv (en), https://arxiv.org/abs/2609.36705 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c4beee58e5b8 (Metraipolis archive dc_c4beee58e5b8, sha256 a9593aa5094548df8e4d844c352ee7a579b20af4273cf768b08f0504edc49a2c)
BibTeX
@misc{dcc4beee58e5b8,
  title = {JudgeProfile: Understanding and Steering Subjectivity in LLM Judges — arXiv:2609.36705},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36705},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c4beee58e5b8; sha256 a9593aa5094548df8e4d844c352ee7a579b20af4273cf768b08f0504edc49a2c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36705. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM judges are inherently subjective, often favoring different responses in pairwise comparison when neither option is objectively wrong. To study this subjectivity, we introduce JudgeProfile, a framework that dissects LLM evaluation into perception (how a judge compares two responses across specific attributes like clarity, correctness, and detail) and prioritization (how much each attribute influences the final choice). We curate SubjectiveSet, a dataset of 50,013 response pairs from 17 public data sources, evaluated by 21 LLM judges across 87 attributes. We find a hidden consensus in perception: judges frequently agree on attribute judgments even when their overall choices diverge. Building on this separation, we first characterize each judge's prioritization using attribute weights estimated from its own overall choices. These weights differ across judges even when estimated from the same attribute judgments. We then learn new weights from reference labels to adapt their decisions to a target evaluation standard. Reweighting perceived attributes improves average held-out agreement with reference labels from 66.48% to 71.97%, outperforming fine-tuning and rubric prompting. Our findings show that understanding and steering the subjectivity of LLM judges requires attention not only to what they perceive, but also to how they prioritize it.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.36705 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.528Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).