Metraipolis › archive › dc_41808cb412e4
An Experimental Study of Hybrid Retrieval, NLI and Open-source LLMs for Scientific Claim Verification — arXiv:2611.03093
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.03093
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:20:20.253Z
- SHA-256 of the archived text
e2dea7a7c094176c6118e35deb1e962c3ebd0fe0a0c45fb1c9fc6ccc96945d90- Stable URL of this page
https://metraipolis.com/documento/dc_41808cb412e4
How to cite
An Experimental Study of Hybrid Retrieval, NLI and Open-source LLMs for Scientific Claim Verification — arXiv:2611.03093 — arXiv (en), https://arxiv.org/abs/2611.03093 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_41808cb412e4 (Metraipolis archive dc_41808cb412e4, sha256 e2dea7a7c094176c6118e35deb1e962c3ebd0fe0a0c45fb1c9fc6ccc96945d90)
BibTeX
@misc{dc41808cb412e4,
title = {An Experimental Study of Hybrid Retrieval, NLI and Open-source LLMs for Scientific Claim Verification — arXiv:2611.03093},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.03093},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_41808cb412e4; sha256 e2dea7a7c094176c6118e35deb1e962c3ebd0fe0a0c45fb1c9fc6ccc96945d90},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03093. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Claim verification is the task of deciding whether a source text supports a claim, contradicts it, or lacks sufficient evidence. Natural Language Inference (NLI) models can judge whether a single sentence supports or contradicts a claim, but they struggle when the evidence is spread across several sentences. Conversely, Large Language Models (LLMs) can reason across sentences, but may draw conclusions the text does not support. In this experimental study, we test whether a conservative fusion rule, labeling a claim Support or Contradict only when both models agree and defaulting to Not Enough Evidence otherwise, combines their strengths. We construct a benchmark of 2,035 claims from SciFact and CliniFact (858 Support, 152 Contradict, 1,025 Not Enough Evidence). We retrieve the top five most relevant sentences per claim using a hybrid of semantic and lexical (BM25) retrieval, which slightly outperforms either method alone. These retrieved sentences are then provided to every verification model. We then compare a DeBERTa-v3-base-mnli-fever-anli NLI baseline against Llama 3.1 8B, Qwen3 8B, and Mistral 7B. Qwen3 achieves the highest Macro-F1 (0.7937), followed by Llama 3.1 (0.7204) and Mistral (0.6388), while the NLI baseline reaches only 0.3157. Contrary to our hypothesis, fusion degrades performance, dropping Macro-F1 to 0.7392, 0.6730, and 0.5991, respectively. To understand this decline, we analyze agreements, disagreements, and common error patterns between the NLI and LLM verifiers. Our error analysis shows that NLI and LLM disagreements often result in correct LLM predictions being overturned. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.03093 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.737Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).