Metraipolis › archive › dc_1901c2bad820

When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation — arXiv:2609.12718

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.12718
License (as archived)
CC0-1.0
Archived
2026-10-16T03:02:53.347Z
SHA-256 of the archived text
e797afb58bc8ed339969dc6a40cdd43389650d2c1f38a16be884381777b9de38
Stable URL of this page
https://metraipolis.com/documento/dc_1901c2bad820

How to cite

When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation — arXiv:2609.12718 — arXiv (en), https://arxiv.org/abs/2609.12718 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1901c2bad820 (Metraipolis archive dc_1901c2bad820, sha256 e797afb58bc8ed339969dc6a40cdd43389650d2c1f38a16be884381777b9de38)
BibTeX
@misc{dc1901c2bad820,
  title = {When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation — arXiv:2609.12718},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.12718},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1901c2bad820; sha256 e797afb58bc8ed339969dc6a40cdd43389650d2c1f38a16be884381777b9de38},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12718. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hallucinations can undermine clinician trust in LLMs, making it important that evaluation methods capture clinically relevant errors. Rubric-based evaluation has become the leading approach for assessing LLMs in medicine, but it is unclear whether rubric scores reflect such errors. We first study this in a controlled setting using MedHallu, finding that more specific rubrics better distinguish correct from hallucinated responses. To test this systematically, we develop a taxonomy of medical hallucination types and a clinician-validated error-injection pipeline that creates matched correct and error-injected responses. Across HealthBench, HealthBench Professional, and LiveMedBench, our clinically relevant hallucinations are missed by rubrics, often leaving scores unchanged. We find that rubrics are most effective when explicitly checking facts, and are less effective for additional or unexpected errors they do not anticipate. A preliminary retrieval-based factuality check recovers some of the rubric-blind errors, suggesting a complementary approach. These findings reveal systematic blind spots in current medical evaluation of LLMs and suggest that rubric scores alone are insufficient to establish clinical reliability, potentially undermining clinician trust and confidence in clinical deployment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.12718 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T02:51:27.815Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).