Metraipolis › archive › dc_e4f35378235f

Evaluating Safety-Critical Communication Behavior of Large Language Models Using Workflow-Embedded Multi-Agent Clinical Simulation (2026) · DOI 10.21203/rs.3.rs-10989591/v1

Source
Metraipolis archive
Original
https://doi.org/10.21203/rs.3.rs-10989591/v1
License (as archived)
CC-BY-4.0
Archived
2026-10-08T05:41:09.789Z
SHA-256 of the archived text
80384f788d453423459ff58176d9f8221a6a839699883b09abd34bb7506d3a1b
Stable URL of this page
https://metraipolis.com/documento/dc_e4f35378235f

How to cite

Evaluating Safety-Critical Communication Behavior of Large Language Models Using Workflow-Embedded Multi-Agent Clinical Simulation (2026) · DOI 10.21203/rs.3.rs-10989591/v1 — Metraipolis archive, https://doi.org/10.21203/rs.3.rs-10989591/v1 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_e4f35378235f (Metraipolis archive dc_e4f35378235f, sha256 80384f788d453423459ff58176d9f8221a6a839699883b09abd34bb7506d3a1b)
BibTeX
@misc{dce4f35378235f,
  title = {Evaluating Safety-Critical Communication Behavior of Large Language Models Using Workflow-Embedded Multi-Agent Clinical Simulation (2026) · DOI 10.21203/rs.3.rs-10989591/v1},
  howpublished = {Metraipolis archive},
  url = {https://doi.org/10.21203/rs.3.rs-10989591/v1},
  note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_e4f35378235f; sha256 80384f788d453423459ff58176d9f8221a6a839699883b09abd34bb7506d3a1b},
  year = {2026}
}

Archived text

Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://doi.org/10.21203/rs.3.rs-10989591/v1. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Articolo recente con licenza aperta: scheda dai metadati Crossref (posted-content).
Titolo: Evaluating Safety-Critical Communication Behavior of Large Language Models Using Workflow-Embedded Multi-Agent Clinical Simulation
Autori: David Power, Theresa Power
Pubblicato: 2026-09-11 · editore: Springer Science and Business Media LLC
DOI: https://doi.org/10.21203/rs.3.rs-10989591/v1
Licenza dell'articolo: CC-BY-4.0

Abstract (testo degli autori, CC-BY-4.0):
Abstract Objective: To evaluate the safety-critical escalation behavior of large language models (LLMs) within a realistic ward-to-registrar communication workflow using a role locked, multi-agent simulation framework, and to assess the impact of structured ISBAR handover on escalation safety and actionability. Materials and Methods: We conducted a controlled, paired simulation study using a role-locked multi agent LLM framework representing ward nurses, registrars (a senior resident physician in specialty training), and nurse managers. Synthetic ward deteri oration case cards defined ground-truth clinical information. For each case, matched escalation transcripts were generated under two conditions: unstruc tured (control) and ISBAR-structured nurse handover. Registrar responses were constrained to produce structured plans containing assessment, actions, timeframes, and escalation triggers. Transcripts were evaluated using a rubric based automated auditing system to assess safety-critical omissions, escalation appropriateness, actionability, closed-loop communication, and hallucinations. A subset underwent independent review by experienced intensive care nurses to evaluate agreement between human and automated assessments. 1 Results: ISBAR-structured handover significantly reduced the frequency of hallucinated content and improved escalation efficiency but did not reduce safety-critical omissions. Across 200 paired (n=400) escalation simulations, ISBAR-structured handover reduced hallucination frequency (10.0% vs 26.5%) and shortened esca lation dialogues (median 7 vs 9 turns) but did not reduce safety-critical omissions (16.0% vs 11.5%). Structured handover improved specification of review time frames and escalation triggers and supported clearer closed-loop confirmation. Human reviewers demonstrated moderate agreement for omission detection and several communication quality measures. Agreement between the automated evaluator and human consensus was limited across several domains, with the automated evaluator showing high sensitivity but low specificity for omission detection and systematically differing from human assessments of actionability and overall safety risk. Discussion: Embedding LLMs within a role-locked escalation workflow reveals safety-critical behaviors that are not observable using static evaluation methods. Structured ISBAR communication improves LLM escalation performance by constraining information flow and reducing hallucinated content, but does not eliminate all failure modes. Human validation identified important differences between automated and expert assessment, suggesting that current LLM-based eval uators may prioritize communication structure and fluency differently from clinicians assessing clinical completeness and safety. Workflow-embedded simu lation offers a scalable approach to pre-deployment evaluation and governance of LLMs in safety-critical clinical communication, while supporting continued expert oversigh...

Fonte: https://api.crossref.org/works/10.21203%2Frs.3.rs-10989591%2Fv1 — metadati Crossref, consultati il 2026-10-07. Interrogazione: «multi-agent systems large language models».