Metraipolis › archive › dc_df660aaae998

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification — arXiv:2609.30467

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.30467
License (as archived): CC0-1.0
Archived: 2026-10-08T02:03:55.902Z
SHA-256 of the archived text: 9a03e1f2c4d5d8e44b2becc9ea53d8b0310b2d1442f68866aa3eb6bcc760c149
Stable URL of this page: https://metraipolis.com/documento/dc_df660aaae998

How to cite

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification — arXiv:2609.30467 — arXiv (en), https://arxiv.org/abs/2609.30467 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_df660aaae998 (Metraipolis archive dc_df660aaae998, sha256 9a03e1f2c4d5d8e44b2becc9ea53d8b0310b2d1442f68866aa3eb6bcc760c149)

BibTeX

@misc{dcdf660aaae998,
  title = {Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification — arXiv:2609.30467},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.30467},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_df660aaae998; sha256 9a03e1f2c4d5d8e44b2becc9ea53d8b0310b2d1442f68866aa3eb6bcc760c149},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30467. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Retrieval-based factuality evaluation, where LLM-generated claims are verified against evidence from authoritative medical corpora, has become the dominant paradigm for scalable hallucination detection in high-stakes clinical settings. Despite the urgency of reliable and transparent medical fact verification, most systems measure performance with aggregate metrics like F1, which obscure where and why failures occur. Existing RAG diagnostics require gold answers or annotated gold evidence, neither of which exists in this regime. We introduce two comprehensive taxonomies, grounded in a case study on the open-ended MedExpert dataset and 3 closed-ended datasets, decomposing failures into retrieval-stage errors along five quality dimensions, and verifier-reasoning errors into six consecutive steps. We adapt an automatic pattern induction pipeline using LLM-as-Judge to label evidence quality and classify verifier reasoning errors at scale, and then stress-test our findings across 4 retrieval methods and 6 frontier verifier models. Our analysis reveals that scaling model size, adding reasoning effort, expanding to authoritative web sources, and applying medical fine-tuning do not resolve these failure modes, demonstrating that they represent fundamental limitations of the retrieve-then-verify paradigm in open-ended medical settings rather than artifacts of outdated systems. We release our code and data at this https URL for the full reproducibility of our results.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2609.30467 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:38:49.426Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).