Metraipolis › archive › dc_dad1deb023eb

From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines — arXiv:2611.03725

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03725
License (as archived)
CC0-1.0
Archived
2026-11-04T18:33:40.917Z
SHA-256 of the archived text
680ddc5c8f2c19a0e4f666866718eeded50e8a97b2c246b38447a6d7875c7dd5
Stable URL of this page
https://metraipolis.com/documento/dc_dad1deb023eb

How to cite

From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines — arXiv:2611.03725 — arXiv (en), https://arxiv.org/abs/2611.03725 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dad1deb023eb (Metraipolis archive dc_dad1deb023eb, sha256 680ddc5c8f2c19a0e4f666866718eeded50e8a97b2c246b38447a6d7875c7dd5)
BibTeX
@misc{dcdad1deb023eb,
  title = {From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines — arXiv:2611.03725},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03725},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dad1deb023eb; sha256 680ddc5c8f2c19a0e4f666866718eeded50e8a97b2c246b38447a6d7875c7dd5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03725. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Retrieval-Augmented Generation (RAG) pipelines are widely used for domain-specific question answering, yet locating faults remains manual and time-consuming. Fault localization is challenging because RAG pipelines consist of interdependent stages, where faults can propagate downstream, masking their origin, or remain undetected even when the final answer is correct. Existing evaluation approaches often focus on individual stages and rely on reference answers or code instrumentation. We present RAGLog, which detects and localizes faults across RAG stages using execution logs, without requiring reference answers or code instrumentation. RAGLog transforms heterogeneous API logs into a common RAG schema and combines rule-based, embedding-based, and LLM-based checks to identify faulty stages. We introduce a live fault injector and RAGLog Faults, a dataset of 3,134 runs from five RAG pipelines with stage-level fault labels. With Claude Haiku 5.5, RAGLog achieves a pooled F1 score of 0.73 on runs with naturally occurring faults, outperforming Claude Code (0.58) and Codex (0.52), and comparable effectiveness on runs with injected faults (0.84 versus 0.80 and 0.82, respectively). It analyzes runs in approximately five seconds at US\$0.005 each, with overall lower cost and latency than agent baselines. RAGLog is available as an open-source tool for debugging, evaluating, and monitoring RAG pipelines.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.03725 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:37.239Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).