Metraipolis › archive › dc_36deeeeb247d

Towards Multi-Method Evaluation of AI in Health: Integrating Benchmarks, Autoraters, and Expert Evaluation — arXiv:2611.05127

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.05127
License (as archived): CC0-1.0
Archived: 2026-11-04T12:36:29.423Z
SHA-256 of the archived text: ef9063d8fbd701854abab6816d63eaf3168a90dbfebc867a89681a9911756b49
Stable URL of this page: https://metraipolis.com/documento/dc_36deeeeb247d

How to cite

Towards Multi-Method Evaluation of AI in Health: Integrating Benchmarks, Autoraters, and Expert Evaluation — arXiv:2611.05127 — arXiv (en), https://arxiv.org/abs/2611.05127 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_36deeeeb247d (Metraipolis archive dc_36deeeeb247d, sha256 ef9063d8fbd701854abab6816d63eaf3168a90dbfebc867a89681a9911756b49)

BibTeX

@misc{dc36deeeeb247d,
  title = {Towards Multi-Method Evaluation of AI in Health: Integrating Benchmarks, Autoraters, and Expert Evaluation — arXiv:2611.05127},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05127},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_36deeeeb247d; sha256 ef9063d8fbd701854abab6816d63eaf3168a90dbfebc867a89681a9911756b49},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05127. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

General-purpose generative AI (GenAI) models are being rapidly adopted in healthcare, but they are challenging to assess with traditional evaluation frameworks. These conceptual, methodological, and practical challenges parallel decades of work in evaluating clinician and health system quality and safety. We describe the adaptation of methods from several fields, combined with emerging technical approaches, into a structured, multi-method evaluation framework designed to give well-rounded assessments of GenAI in health at scale. The approach first anchors evaluation to Critical User Journeys (CUJs) to help prioritize out of a large choice space, and maps these to a phased AI system lifecycle (base model development, product development, and post-deployment monitoring), since evaluation at each stage requires different methodological approaches. In this context, the approach then integrates three distinct evaluation methods (closed-question benchmarks, LLM-as-a-judge autoraters, and expert human review) across three core domains: accuracy, safety, and usefulness. Taken together, this approach provides a strategic framework for bridging the gap between benchmarking results and performance in the real world with the goal of supporting decision-making and accountability across development, procurement, and deployment for product teams, health systems, and regulators.

Tema degli esploratori: governance digitale e beni comuni (elenco arXiv cs.CY, nuovi lavori del giorno).
Categorie arXiv: Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2611.05127 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:28:02.115Z dalla pagina https://arxiv.org/list/cs.CY/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).