Metraipolis › archive › dc_79dd7ecc6685

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions — arXiv:2610.09229

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.09229
License (as archived)
CC0-1.0
Archived
2026-10-08T11:10:44.607Z
SHA-256 of the archived text
015dd16615cfe3765244131288acecea1471232dd655f63068ae811e3d0a184b
Stable URL of this page
https://metraipolis.com/documento/dc_79dd7ecc6685

How to cite

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions — arXiv:2610.09229 — arXiv (en), https://arxiv.org/abs/2610.09229 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_79dd7ecc6685 (Metraipolis archive dc_79dd7ecc6685, sha256 015dd16615cfe3765244131288acecea1471232dd655f63068ae811e3d0a184b)
BibTeX
@misc{dc79dd7ecc6685,
  title = {Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions — arXiv:2610.09229},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09229},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_79dd7ecc6685; sha256 015dd16615cfe3765244131288acecea1471232dd655f63068ae811e3d0a184b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09229. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We introduce \textbf{Conditional Accuracy Profiling} (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality. CAP is benchmark-agnostic: it can be applied directly when a benchmark provides the required annotations, approximately through task-subset proxies, or through controlled augmentation when perturbation pairs can be generated. We instantiate CAP on seven LLM judges across six pairwise judging benchmarks, including \textsc{judgerEva-Standard}, a controlled testbed we created to support all eight conditions. CAP exposes profile differences hidden by aggregate accuracy: on \textsc{judgerEva}'s judge-independent Hard-Constructed subset, the two judges most sensitive to omitted qualifications rank in the bottom three of seven by overall accuracy, so omission sensitivity is not predicted by aggregate accuracy. Across benchmarks, Position Robustness shows the strongest rank stability (mean Spearman $\bar{\rho}{=}0.87$) but is itself fragile under JudgeBench-Pro adversarial stress, showing the largest mean accuracy drop among the shared conditions, though the dominant degradation channel varies by judge. Condition-level profiles provide a more actionable basis than aggregate accuracy for selecting LLM judges.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.09229 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:06.063Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).