Metraipolis › archive › dc_86ec01674c2b
Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries — arXiv:2608.26385
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.26385
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T14:25:03.759Z
- SHA-256 of the archived text
faee70027914aa79867ca23dcc4bb9ecf4a6f5f5060c7114a9776c820d8d6ffb- Stable URL of this page
https://metraipolis.com/documento/dc_86ec01674c2b
How to cite
Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries — arXiv:2608.26385 — arXiv (en), https://arxiv.org/abs/2608.26385 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_86ec01674c2b (Metraipolis archive dc_86ec01674c2b, sha256 faee70027914aa79867ca23dcc4bb9ecf4a6f5f5060c7114a9776c820d8d6ffb)
BibTeX
@misc{dc86ec01674c2b,
title = {Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries — arXiv:2608.26385},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.26385},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_86ec01674c2b; sha256 faee70027914aa79867ca23dcc4bb9ecf4a6f5f5060c7114a9776c820d8d6ffb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26385. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Volume-based accuracy rewards retrieval-augmented generation (RAG) systems for guessing: a system that answers everything outscores one that declines when its knowledge base cannot support an answer. Building on the confidence-target analysis of Kalai et al. (2025), we present a penalty-aware evaluation framework for deployed RAG products, combining (i) asymmetric scoring (correct +1, wrong -4, abstain 0), (ii) knowledge-gap canaries, questions whose answers are verifiably absent from the knowledge base, so that any answer constitutes ungrounded generation from parametric memory, and (iii) a failure-attribution pipeline that separates retrieval, generation, and abstention-policy failures. Applying the framework to three commercial RAG systems and a no-retrieval baseline on SimpleQA-Verified (1,000 questions x 3 repeats, graded blind by a cross-family three-judge panel with 98.9% unanimity), we find that accuracy when answering is closely clustered across systems (97.0-98.0%), while canary violation rates differ roughly sixfold (16.7% vs. 98.1%). The systems are separated less by what they answer correctly than by whether they answer at all when they should not, and penalty-aware scoring reorders the volume-based ranking accordingly; the reordering is stable across penalty settings from k=1 to k=9. All code, configurations, transcripts, and judge votes are released for independent audit. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.26385 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T13:16:53.633Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).