Metraipolis › archive › dc_7d7c01dadcc9
Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks — arXiv:2609.16023
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.16023
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T19:19:05.435Z
- SHA-256 of the archived text
c679e33bd3f3fb508e69d4c58575dd0382d5180ed3e131173b50060ecf38ead2- Stable URL of this page
https://metraipolis.com/documento/dc_7d7c01dadcc9
How to cite
Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks — arXiv:2609.16023 — arXiv (en), https://arxiv.org/abs/2609.16023 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d7c01dadcc9 (Metraipolis archive dc_7d7c01dadcc9, sha256 c679e33bd3f3fb508e69d4c58575dd0382d5180ed3e131173b50060ecf38ead2)
BibTeX
@misc{dc7d7c01dadcc9,
title = {Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks — arXiv:2609.16023},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.16023},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d7c01dadcc9; sha256 c679e33bd3f3fb508e69d4c58575dd0382d5180ed3e131173b50060ecf38ead2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16023. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to two clinical benchmarks (HealthBench Professional and LiveMedBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of criteria as non-atomic and 65.4% as misaligned/rigid. Then, we show that these flaws are meaningful and not simply cosmetic. As an example, rewriting bundled criteria of the form "at least one of / all of the following" as equally weighted children and regrading identical responses shifts scores by up to 15.9 percentage points on affected conversations, with disjunctive bundles inflating scores and conjunctive bundles deflating them. We also find that RIFT generally under-detects bundling on clinical rubrics, flagging 3.3% of LiveMedBench criteria as non-atomic where surface-form analysis finds structure in 25.8%. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.16023 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T18:32:32.792Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).