Metraipolis › archive › dc_ef8e43afd063

Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets — arXiv:2611.02484

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02484
License (as archived)
CC0-1.0
Archived
2026-11-03T11:02:41.699Z
SHA-256 of the archived text
229c4d7340c317c82ce0dc638d4f30d4b58d5af50aff220104ede8c70e9a02b9
Stable URL of this page
https://metraipolis.com/documento/dc_ef8e43afd063

How to cite

Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets — arXiv:2611.02484 — arXiv (en), https://arxiv.org/abs/2611.02484 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef8e43afd063 (Metraipolis archive dc_ef8e43afd063, sha256 229c4d7340c317c82ce0dc638d4f30d4b58d5af50aff220104ede8c70e9a02b9)
BibTeX
@misc{dcef8e43afd063,
  title = {Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets — arXiv:2611.02484},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02484},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef8e43afd063; sha256 229c4d7340c317c82ce0dc638d4f30d4b58d5af50aff220104ede8c70e9a02b9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02484. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM agents are increasingly built into data-analysis software, yet little evidence shows how reliably they preprocess clinical data or whether the tools they invoke give correct, reproducible output. We report a benchmark of THReD, a web-based system that turns electronic health record (EHR) extracts into AI-ready datasets through a chat agent whose proposed steps run only after user confirmation. The agent benchmark has 141 tasks (100 templates) in seven categories, including goal-level requests, failure recovery, requests calling for no action, schema variation and prompt injection through the data, on synthetic tables calibrated to a regional health information exchange; success is judged by checks on the produced data. Over 2,550 episodes with Claude Haiku 5.5 in four context configurations, the deployed agent completed 92.7% (95% CI 88.1-96.4) of tasks on its first answer, abstained on every advice request, recovered from every injected fault, and never proposed an action requested by planted text. Without the data profile, completion fell to 58.0%; goal-level requests were hardest (80.6%). Six experiments on real records of two chronic-disease cohorts and on synthetic data at scale validate the tools: 91-92% of inferred patient classes and 99.7-99.8% of inferred vital-sign labels were correct in masked reconstruction; a dictionary builder combining curated dictionaries with LLM suggestions mapped 99.0% of 2,208 raw variants correctly; all 2,176 replays of 128 saved pipelines gave byte-identical files under pinned dependencies; subgroup differences in inference accuracy vanished within facility and lookup key; and the same-day consolidation rule changed 90-day emergency revisit rates by up to 3.2 points without shifting cohort composition by more than one point. Processing time grew roughly linearly to 5.5 million records while the prompt stayed at about 13 thousand tokens.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02484 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:45.821Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).