Metraipolis › archive › dc_2d2ebaf94d3d
Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment — arXiv:2609.11185
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.11185
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T05:23:44.916Z
- SHA-256 of the archived text
1402d96b1524a5c9f7cff20be04477e07f325ddc310070b2f513469a865cd201- Stable URL of this page
https://metraipolis.com/documento/dc_2d2ebaf94d3d
How to cite
Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment — arXiv:2609.11185 — arXiv (en), https://arxiv.org/abs/2609.11185 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2d2ebaf94d3d (Metraipolis archive dc_2d2ebaf94d3d, sha256 1402d96b1524a5c9f7cff20be04477e07f325ddc310070b2f513469a865cd201)
BibTeX
@misc{dc2d2ebaf94d3d,
title = {Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment — arXiv:2609.11185},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.11185},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2d2ebaf94d3d; sha256 1402d96b1524a5c9f7cff20be04477e07f325ddc310070b2f513469a865cd201},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.11185. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evidence-based medicine demands strict logical consistency, yet current evaluations of large language models (LLMs) prioritize superficial label matching over genuine reasoning. We introduce LogiMed-RoB, a benchmark grounded in Cochrane Risk of Bias (RoB) 2.0 expert logic, comprising 860 randomized controlled trials (RCTs) and 14,820 queries. It evaluates models under the Hierarchical Logical Consistency (HLC) framework across four dimensions: Atomic Consistency, Domain Consistency, Aggregation Consistency, and Evidential Faithfulness. Experiments on 10 state-of-the-art LLMs reveal a catastrophic Error Compounding Effect: despite the top model reaching 98.88% Atomic Consistency, its end-to-end consistency collapses to 45.13%, with several open-weight architectures plummeting to nearly 0%. We further uncover a systematic evidence-reasoning gap: even when models retrieve high-quality evidence, they fail to deduce correct outcomes in 18.63-40.05% of cases, while Blind Guess Rates reach 48.28%. LogiMed-RoB demonstrates that high outcome accuracy can conceal critical reasoning flaws, underscoring the necessity of white-box logical verification for clinical deployment. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-11 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.11185 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T05:17:51.982Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).