Metraipolis › archive › dc_fbb6444e6def
A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench — arXiv:2608.12138
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.12138
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T05:16:28.337Z
- SHA-256 of the archived text
072e24d3e0785618c73090da9b7788554362a8decc92200a628e9823e234492e- Stable URL of this page
https://metraipolis.com/documento/dc_fbb6444e6def
How to cite
A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench — arXiv:2608.12138 — arXiv (en), https://arxiv.org/abs/2608.12138 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbb6444e6def (Metraipolis archive dc_fbb6444e6def, sha256 072e24d3e0785618c73090da9b7788554362a8decc92200a628e9823e234492e)
BibTeX
@misc{dcfbb6444e6def,
title = {A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench — arXiv:2608.12138},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.12138},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbb6444e6def; sha256 072e24d3e0785618c73090da9b7788554362a8decc92200a628e9823e234492e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12138. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings. We evaluate VITA, a retrieval-augmented generation (RAG) system purpose-built for contextual knowledge retrieval in India and other low- and middle-income (LMIC) settings. VITA retrieves from a curated corpus of disease-specific guidelines, India-specific antimicrobial resistance data, national formulary constraints, and resource-limited care protocols; its architecture and corpus are proprietary, but the benchmark, the physician-written rubrics, and our full response and scoring outputs are public for independent verification. On 4,023 English-language HealthBench questions (80.5% of the benchmark), scored with a GPT-4.1 judge, VITA ranked first with 51.9% of possible rubric points, ahead of GPT-5.4 (46.1%), o4-mini (44.3%), Gemini 3.1 Pro (42.6%), and Claude Sonnet 4.6 (37.3%), and scored highest on 45.4% of questions. To test robustness to newer models and judge lineage, a 500-question subset was re-run against current-generation models (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Pro, Grok 4.3) and graded by a neutral open-weight judge (DeepSeek-V4-Pro) sharing no lineage with any system tested. Here the gap narrowed to parity: VITA and GPT-5.5 were statistically indistinguishable on mean per-question score, while VITA led on points-weighted score and won the most questions. VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower. These results indicate that a purpose-built clinical RAG system remains competitive with frontier LLMs on an open benchmark, consistent with corpus specificity as a design variable that improves grounding at some cost to communication polish. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Information Retrieval (cs.IR); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.12138 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:08:46.766Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).