Metraipolis › archive › dc_bcfb34bfdab0

What Next-Event Accuracy Cannot See: Closed-Loop Evaluation of Emergency Department Trajectory Simulators — arXiv:2609.31635

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.31635
License (as archived)
CC0-1.0
Archived
2026-10-07T20:59:08.626Z
SHA-256 of the archived text
7765055533235f811485ed5e1a1a05d4d74d6f05a1eb57f5608adbc87d75caef
Stable URL of this page
https://metraipolis.com/documento/dc_bcfb34bfdab0

How to cite

What Next-Event Accuracy Cannot See: Closed-Loop Evaluation of Emergency Department Trajectory Simulators — arXiv:2609.31635 — arXiv (en), https://arxiv.org/abs/2609.31635 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bcfb34bfdab0 (Metraipolis archive dc_bcfb34bfdab0, sha256 7765055533235f811485ed5e1a1a05d4d74d6f05a1eb57f5608adbc87d75caef)
BibTeX
@misc{dcbcfb34bfdab0,
  title = {What Next-Event Accuracy Cannot See: Closed-Loop Evaluation of Emergency Department Trajectory Simulators — arXiv:2609.31635},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31635},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bcfb34bfdab0; sha256 7765055533235f811485ed5e1a1a05d4d74d6f05a1eb57f5608adbc87d75caef},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31635. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Clinical trajectory models are usually evaluated by next-event accuracy on observed histories. Simulation is different: models must condition on their own generated events, allowing errors to compound. Although this problem is well known in sequence modelling, it has not been systematically quantified for clinical trajectory simulators. We developed EDSim-Bench to evaluate this failure mode using 425,028 MIMIC-IV-ED stays, with external replication on MC-MED, and release the evaluation protocol and scoring code. Starting from held-out visit prefixes, models generate the remainder of each visit and are evaluated on termination, event composition, timing, conditional fidelity, and occupancy forecasting, with a train-only order-3 n-gram as a reference baseline. Despite next-event accuracies within 0.001, three neural architectures behaved very differently under rollout. Across seeds, one Transformer recipe ranged from 0.43 to 0.96 in termination score and from 4.2- to 137-fold the divergence of the n-gram; no prefix-trained neural model approached the n-gram on termination or event composition. Inference-time interventions improved termination but did not jointly recover composition and timing. Supervising every eligible sequence position rather than only the final prefix position was associated with one to two orders of magnitude lower divergence across Transformer, GRU, and LSTM models, with the pattern persisting under model scaling, temporal shift, and external-site evaluation. Nevertheless, even the best model generated visits approximately half as long as observed, and model rankings reversed on occupancy forecasting, a downstream quantity relevant to bed management. These results show that next-event accuracy is insufficient to evaluate clinical trajectory simulators and motivate closed-loop evaluation across seeds, rollout criteria, and downstream tasks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.31635 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.958Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).