Metraipolis › archive › dc_92773dde02ba
SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence — arXiv:2608.29582
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.29582
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:47:55.620Z
- SHA-256 of the archived text
b4352e364c348caacf6ecc072c6fc0fe2bc040609a4609d73f46dc68bb00adba- Stable URL of this page
https://metraipolis.com/documento/dc_92773dde02ba
How to cite
SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence — arXiv:2608.29582 — arXiv (en), https://arxiv.org/abs/2608.29582 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_92773dde02ba (Metraipolis archive dc_92773dde02ba, sha256 b4352e364c348caacf6ecc072c6fc0fe2bc040609a4609d73f46dc68bb00adba)
BibTeX
@misc{dc92773dde02ba,
title = {SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence — arXiv:2608.29582},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.29582},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_92773dde02ba; sha256 b4352e364c348caacf6ecc072c6fc0fe2bc040609a4609d73f46dc68bb00adba},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29582. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Current evaluations of large language models (LLMs) primarily focus on factual knowledge retrieval, overlooking the fundamental challenge of navigating the complex, non-bijective mappings between clinical indicators and diagnoses. Existing benchmarks fail to assess whether large language models truly possess the reasoning capability required for diagnostic ambiguity scenarios, where identical clinical presentations may correspond to different etiologies, and diagnostic convergence scenarios, where heterogeneous symptoms ultimately indicate the same disease. To address this issue, we propose SUP-MIMIC, a multi-task framework utilizing MIMIC-IV-v3.1 that comprises Basic Assessment (BA), Diagnostic Divergence Task (DDT), and Diagnostic Convergence Task (DCT). Specifically, DDT is designed to evaluate the model's "one-to-many" disambiguation capability among phenotypically similar cases, while DCT assesses the model's ability to identify "many-to-one" diagnostic patterns across different pathophysiological pathways. Comprehensive evaluation of state-of-the-art LLMs reveals substantial performance degradation on DDT and DCT compared to baseline tasks, exposing a systemic reliance on statistical shortcuts over genuine causal reasoning. Our findings further highlight a conservative bias toward "healthy" predictions, implying non-trivial risks for missed diagnoses in realistic medical settings. This work establishes a rigorous methodology for quantifying clinical reasoning robustness and provides a roadmap for enhancing the safety of language models in clinical medicine. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.29582 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.782Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).