Metraipolis › archive › dc_d97c162b8f79
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations — arXiv:2609.15855
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15855
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T22:54:07.883Z
- SHA-256 of the archived text
19190036574866588c351efe6c05526bd74632c58ac2e90d9301d6a16228b248- Stable URL of this page
https://metraipolis.com/documento/dc_d97c162b8f79
How to cite
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations — arXiv:2609.15855 — arXiv (en), https://arxiv.org/abs/2609.15855 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d97c162b8f79 (Metraipolis archive dc_d97c162b8f79, sha256 19190036574866588c351efe6c05526bd74632c58ac2e90d9301d6a16228b248)
BibTeX
@misc{dcd97c162b8f79,
title = {K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations — arXiv:2609.15855},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15855},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d97c162b8f79; sha256 19190036574866588c351efe6c05526bd74632c58ac2e90d9301d6a16228b248},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15855. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk conversations remains poorly characterised. We developed K-Bench, a clinician-calibrated, protected benchmark evaluating 125 model configurations representing 33 base models from 14 providers across a fixed cohort of 200 multi-turn vignettes involving suicide, self-harm, domestic violence, substance misuse, and no-risk presentations. Synthetic patient conversations showed substantial distributional overlap with real human-AI conversations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible item comparisons from 151 clinician-rated transcripts. Leading models combined strong supportive conversation with combined-risk scores above 95, whereas risk exploration exposed substantial variation among lower-performing configurations. Therapeutic prompting produced configuration-specific gains concentrated among weaker models, while elevated reasoning produced no average improvement. K-Bench combines broader clinical coverage and configuration-scale comparison with a continuously updated public leaderboard whose operational test materials are protected from direct optimisation. The leaderboard is available at this http URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.15855 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:29:03.939Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).