Metraipolis › archive › dc_ea4c7bc567e8

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy — arXiv:2609.38480

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38480
License (as archived)
CC0-1.0
Archived
2026-10-07T12:43:49.182Z
SHA-256 of the archived text
f09da0cd354eb85e022174439a4010d21d73bc5f721451348daf0ec6198ca9ab
Stable URL of this page
https://metraipolis.com/documento/dc_ea4c7bc567e8

How to cite

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy — arXiv:2609.38480 — arXiv (en), https://arxiv.org/abs/2609.38480 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ea4c7bc567e8 (Metraipolis archive dc_ea4c7bc567e8, sha256 f09da0cd354eb85e022174439a4010d21d73bc5f721451348daf0ec6198ca9ab)
BibTeX
@misc{dcea4c7bc567e8,
  title = {KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy — arXiv:2609.38480},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38480},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ea4c7bc567e8; sha256 f09da0cd354eb85e022174439a4010d21d73bc5f721451348daf0ec6198ca9ab},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38480. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment. All the code and data is available on this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.38480 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.889Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).