Metraipolis › archive › dc_da03c2b455f3
The RAT: A Unified Bayesian Model for RAG Evaluation — arXiv:2608.24753
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.24753
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:18:18.848Z
- SHA-256 of the archived text
654c885e778c50d83125c113122b50522a942d24fbced261b8cdc244191fb71d- Stable URL of this page
https://metraipolis.com/documento/dc_da03c2b455f3
How to cite
The RAT: A Unified Bayesian Model for RAG Evaluation — arXiv:2608.24753 — arXiv (en), https://arxiv.org/abs/2608.24753 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_da03c2b455f3 (Metraipolis archive dc_da03c2b455f3, sha256 654c885e778c50d83125c113122b50522a942d24fbced261b8cdc244191fb71d)
BibTeX
@misc{dcda03c2b455f3,
title = {The RAT: A Unified Bayesian Model for RAG Evaluation — arXiv:2608.24753},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.24753},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_da03c2b455f3; sha256 654c885e778c50d83125c113122b50522a942d24fbced261b8cdc244191fb71d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24753. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating that retrieval-success annotations are more informative than task-success annotations for estimating policy adherence, and provide an information-theoretic explanation for this asymmetry. Finally, we extend the model to incorporate LLM-as-a-judge annotations as calibrated noisy observations, enabling practitioners to combine limited human judgments with cheaper automated assessments within a unified probabilistic model. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.24753 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T06:17:29.379Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).