Metraipolis › archive › dc_ede8fbc8c853
Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety — arXiv:2608.02617
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.02617
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:17:25.191Z
- SHA-256 of the archived text
b4ee00f34e58d71c5d576ce455115451f33304f57f99b775f86412b4fa57db35- Stable URL of this page
https://metraipolis.com/documento/dc_ede8fbc8c853
How to cite
Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety — arXiv:2608.02617 — arXiv (en), https://arxiv.org/abs/2608.02617 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ede8fbc8c853 (Metraipolis archive dc_ede8fbc8c853, sha256 b4ee00f34e58d71c5d576ce455115451f33304f57f99b775f86412b4fa57db35)
BibTeX
@misc{dcede8fbc8c853,
title = {Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety — arXiv:2608.02617},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.02617},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ede8fbc8c853; sha256 b4ee00f34e58d71c5d576ce455115451f33304f57f99b775f86412b4fa57db35},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02617. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We evaluate whether clinician pairwise preferences provide a reliable signal of clinical safety in large language model (LLM) evaluation using expert feedback from MOOVE (Massive Open Online Validation and Evaluation), a clinician-led platform collecting blinded pairwise preferences alongside multi-criterion rubric ratings. Clinicians assign scores on a discrete $[-2, +2]$ scale, where negative values indicate clinically unsafe or misleading content. Using 26{,}804 pairwise judgments across outputs from 13 LLMs, contributed by more than 736 clinicians across 28+ countries, we find that clinician preference is a poor proxy for safety-critical performance. Models ranking highly under pairwise preference can still exhibit substantial rates of clinically meaningful failures ($\leq -1$) on dimensions such as \emph{Harmlessness} and \emph{Accuracy}. These failures are unevenly distributed across specialties, creating domain-specific ``no-go zones'' not visible in aggregate rankings or single-number leaderboards. We further analyze contributing factors including prompt length, refusal and escalation behavior, and the relative contributions of safety-critical versus surface-level features. A substantial fraction of preference votes carry no positive safety signal, while feature decomposition shows that surface-level characteristics explain slightly more preference variation than safety-critical rubric differences. Finally, we introduce a clinically adjusted preference ranking combining pairwise preference with rubric-derived feedback, producing a more safety-aware ordering than raw Bradley--Terry strength alone. Our findings support evaluation practices that separate preference from safety, report safety-critical failure rates directly, and incorporate clinically grounded adjustments when ranking LLMs for clinical decision making.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.02617 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:52.891Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).