Metraipolis › archive › dc_626442df2a6f

An ambiguity taxonomy for evaluating large language model performance on clinical registry abstraction: a multi-site prospective study — arXiv:2608.20373

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.20373
License (as archived)
CC0-1.0
Archived
2026-10-18T10:18:12.342Z
SHA-256 of the archived text
df0f879fc978fdf15fcc7cc65ffc426e76e9f8a9764023b9d7afed8ea441340c
Stable URL of this page
https://metraipolis.com/documento/dc_626442df2a6f

How to cite

An ambiguity taxonomy for evaluating large language model performance on clinical registry abstraction: a multi-site prospective study — arXiv:2608.20373 — arXiv (en), https://arxiv.org/abs/2608.20373 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_626442df2a6f (Metraipolis archive dc_626442df2a6f, sha256 df0f879fc978fdf15fcc7cc65ffc426e76e9f8a9764023b9d7afed8ea441340c)
BibTeX
@misc{dc626442df2a6f,
  title = {An ambiguity taxonomy for evaluating large language model performance on clinical registry abstraction: a multi-site prospective study — arXiv:2608.20373},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.20373},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_626442df2a6f; sha256 df0f879fc978fdf15fcc7cc65ffc426e76e9f8a9764023b9d7afed8ea441340c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.20373. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Objective: To evaluate large language model (LLM) performance on unprocessed electronic medical record (EMR) data for clinical registry abstraction. Methods: We evaluated LLM performance answering registry questions for the American College of Cardiology National Cardiovascular Data Registry (ACC NCDR). In a pilot study at an academic medical center, the model identified candidate data sources for each registry question and experienced abstractors used these results to define question-specific document sets. In a validation study at a second center with a second ACC NCDR registry, the LLM answered questions using the question-specific document sets. Before reviewing any output, two abstractors independently established the ground truth and assigned each question to one of six categories, ordered by the ambiguity and clinical reasoning required to resolve it: Medication/Event Flag, Binary Clinical Presence, Administrative, Quantitative Laboratory/Physiologic, Clinical Interpretation, and Event Timing. Results: The analytical sample comprised 9,430 abstractor answers reconciled to 4,715 consensus answers (501 pilot; 4,214 validation). In the pilot, candidate data sources per question averaged between 14.6 (SD 13.9) for demographics and 89.2 (SD 56.1) for history and risk factors. In validation, human inter-rater agreement was approximately 98\% while 87\% of LLM answers exactly matched consensus, 2\% partially, and 9\% did not. Mean question-level accuracy was 91.5\% (SD 13.4\%) across 157 questions with at least 20 answers, and declined as ambiguity increased, from 96\% for Medication/Event Flag to 62\% for Event Timing questions. Conclusions: LLMs answering clinical registry questions on unprocessed EMR data achieved far lower accuracy than human abstractors. LLM accuracy fell steadily as ambiguity and the level of required clinical reasoning increased.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.20373 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T10:17:37.558Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).