Metraipolis › archive › dc_d2e8ad969484
AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research — arXiv:2609.16519
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.16519
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T18:32:06.023Z
- SHA-256 of the archived text
5eb86ee945000bdf79c7da7d3d74f4f7a5d0d1ee32e97a2cf13a7389fd89dcb6- Stable URL of this page
https://metraipolis.com/documento/dc_d2e8ad969484
How to cite
AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research — arXiv:2609.16519 — arXiv (en), https://arxiv.org/abs/2609.16519 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d2e8ad969484 (Metraipolis archive dc_d2e8ad969484, sha256 5eb86ee945000bdf79c7da7d3d74f4f7a5d0d1ee32e97a2cf13a7389fd89dcb6)
BibTeX
@misc{dcd2e8ad969484,
title = {AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research — arXiv:2609.16519},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.16519},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d2e8ad969484; sha256 5eb86ee945000bdf79c7da7d3d74f4f7a5d0d1ee32e97a2cf13a7389fd89dcb6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16519. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Scientific research increasingly relies on large, heterogeneous data sources, motivating interest in retrieval-augmented generation (RAG) systems that provide natural language access to scientific knowledge and research workflows. Researchers are exploring the viability of these systems as natural language interfaces for document search and for generating analysis code and pipeline components. At the same time, concerns about data privacy and control over research infrastructure have motivated interest in open-weight models and open-source deployments hosted within research institutions. In astronomy, this development follows a long history of computational infrastructure development, from archival databases and Structured Query Language (SQL)-based systems to large language model (LLM)-assisted research tools. This paper presents a domain-expert evaluation of faithfulness for AquiLLM, an open-weight, offline RAG-LLM platform designed to support scientific research groups in the use and preservation of tacit and formal knowledge. We define faithfulness as the extent to which generated responses remain grounded in retrieved scientific context without unsupported claims or omissions. We report results from an astronomy case study evaluating AquiLLM across retrieval and scientific analysis tasks. AquiLLM performs most reliably on explicit retrieval-oriented questions grounded in the RAG collection, while faithfulness degrades for queries requiring synthesis or ambiguity resolution. These results highlight both the promise and limitations of open-weight RAG-LLM systems for scientific research and demonstrate the importance of domain-expert evaluation beyond standard benchmark leaderboards. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.16519 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T18:03:09.440Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).