Metraipolis › archive › dc_0010989a5aaa

Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs — arXiv:2609.13582

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.13582
License (as archived): CC0-1.0
Archived: 2026-10-15T22:48:37.818Z
SHA-256 of the archived text: ec89ba83d8e979fc61fa3ad7cbf7f920b6b4b0a4bdea7e73ce859b747ed95b92
Stable URL of this page: https://metraipolis.com/documento/dc_0010989a5aaa

How to cite

Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs — arXiv:2609.13582 — arXiv (en), https://arxiv.org/abs/2609.13582 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0010989a5aaa (Metraipolis archive dc_0010989a5aaa, sha256 ec89ba83d8e979fc61fa3ad7cbf7f920b6b4b0a4bdea7e73ce859b747ed95b92)

BibTeX

@misc{dc0010989a5aaa,
  title = {Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs — arXiv:2609.13582},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.13582},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0010989a5aaa; sha256 ec89ba83d8e979fc61fa3ad7cbf7f920b6b4b0a4bdea7e73ce859b747ed95b92},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13582. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A clinical agent benchmark can report the same verdict on identical inputs while the agent files a materially different order on each run. Such agents order tests, request medications and place referrals, yet benchmarks typically score one run per task and rarely ask whether identical inputs produce identical actions; MedAgentBench, the benchmark we use, scores a single attempt and says so. To measure this gap we introduce "same-input rerun", which replays a task with every input held fixed and compares the orders rather than the score, with six reliability metrics, and apply it to 1000 MedAgentBench runs across 50 tasks from its five write-capable families, two open-weight models below ten billion parameters quantised to four bits, and two temperatures. The study establishes that action-level divergence exists and can pass unrecorded by the score, not that any rate generalises. Under the 8B model at temperature 0.7, all 43 ordering groups emit a different set of orders across five identical runs, 26 emit the order on some runs and not others, and 28 record a different coded value, dose or analyte. In 22 of those 43 the benchmark reports the same failing verdict for materially different behaviour, as it does for all 10 divergent groups of the 4B model at 0.7. Orders also reach different endpoints across runs, one of which the record server rejects while the agent is told it succeeded. These findings motivate repeated-run evaluation, action-level stability reporting and execution-faithful environment feedback in clinical-agent benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.13582 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.521Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).