Metraipolis › archive › dc_1aff17b554dc
Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models — arXiv:2609.16739
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.16739
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T19:19:17.938Z
- SHA-256 of the archived text
852427eb6dca70b294a9c821629be043005cf1d40a7f3aa67819ffcf81a692f5- Stable URL of this page
https://metraipolis.com/documento/dc_1aff17b554dc
How to cite
Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models — arXiv:2609.16739 — arXiv (en), https://arxiv.org/abs/2609.16739 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1aff17b554dc (Metraipolis archive dc_1aff17b554dc, sha256 852427eb6dca70b294a9c821629be043005cf1d40a7f3aa67819ffcf81a692f5)
BibTeX
@misc{dc1aff17b554dc,
title = {Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models — arXiv:2609.16739},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.16739},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1aff17b554dc; sha256 852427eb6dca70b294a9c821629be043005cf1d40a7f3aa67819ffcf81a692f5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16739. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Background: Large language models (LLMs) have achieved physician-comparable performance on multiple-choice medical knowledge examinations, but their capabilities in clinical history taking, urgency assessment, and safety remain insufficiently evaluated. We proposed Japanese Stroke LLM Evaluation, a multi-turn conversational benchmark for stroke care in Japanese, and evaluated LLM performance and safety under practice-oriented conditions. Methods: We created 10 stroke and related-condition cases and evaluated LLMs in multi-turn Japanese conversations. The LLM acted as physician, while a board-certified neurosurgeon acted as simulated patient and evaluator. Each case comprised history-taking and action phases scored using pre-specified criteria. Errors that could directly threaten life were defined as critical mistakes. The safety threshold was at least 80% overall with zero critical mistakes. Eighteen models were evaluated in October 2025 and June 2026. Results: Claude Fable 5 achieved the highest score (87.4%) with zero critical mistakes, followed by Claude Opus 4.7 (80.3%) and GLM-5.2 (75.6%). Two leaders met the safety threshold. Eleven models made 17 critical mistakes, including failure to confirm laboratory results or blood glucose before t-PA, surgery before airway stabilization, omission of cervical vascular evaluation, and t-PA outside its indication. History-taking question count correlated with history-taking score (r = 0.648, p = 0.007). Conclusions: Japanese Stroke LLM Evaluation provides a benchmark for LLM performance under practice-oriented conditions, including a cap on history-taking questions. Cases and evaluations were created by neurosurgical specialists rather than using an LLM-as-judge approach. Performance improved across cloud-based and on-premise models in 2026, with some exceeding the safety threshold. Further evaluation using real-world cases is required. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.16739 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T18:32:32.825Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).