Metraipolis › archive › dc_285dc0613070

A Living Benchmark for Information Retrieval from Electronic Health Records — arXiv:2609.30205

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.30205
License (as archived): CC0-1.0
Archived: 2026-10-08T04:26:32.122Z
SHA-256 of the archived text: 2659aaa4b30ca7c91e3c4ca0ac5049d11e440e041237f4895ca96adf6e0b301a
Stable URL of this page: https://metraipolis.com/documento/dc_285dc0613070

How to cite

A Living Benchmark for Information Retrieval from Electronic Health Records — arXiv:2609.30205 — arXiv (en), https://arxiv.org/abs/2609.30205 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_285dc0613070 (Metraipolis archive dc_285dc0613070, sha256 2659aaa4b30ca7c91e3c4ca0ac5049d11e440e041237f4895ca96adf6e0b301a)

BibTeX

@misc{dc285dc0613070,
  title = {A Living Benchmark for Information Retrieval from Electronic Health Records — arXiv:2609.30205},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.30205},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_285dc0613070; sha256 2659aaa4b30ca7c91e3c4ca0ac5049d11e440e041237f4895ca96adf6e0b301a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30205. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements. We present a scalable framework that automatically generates question--answer pairs from longitudinal EHR notes. Nineteen clinicians validate the benchmark generator, producing the Benchmark for Retrieving Information in EHRs (BRIE), a continuously maintainable evaluation dataset. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, particularly for questions requiring synthesis across multiple documents and encounters. Because the generator itself is validated, BRIE supports evaluations that static benchmarks cannot, including the generation of multiple answers that reflect variation in clinician reasoning for robust performance assessment and continuously refreshing benchmark content to guard against leakage. Our results demonstrate that scalable benchmark generation enables rigorous, up-to-date evaluation of clinical LLMs as they are deployed in rapidly evolving healthcare settings.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.30205 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T03:50:57.896Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).