Metraipolis › archive › dc_f04064299618

From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines — arXiv:2611.03725

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03725
License (as archived)
CC0-1.0
Archived
2026-11-04T18:22:00.922Z
SHA-256 of the archived text
439b2d1b0209f33dadd48eb702ca0279b0a8ae530d9c61526ebbe900dfa2b667
Stable URL of this page
https://metraipolis.com/documento/dc_f04064299618

How to cite

From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines — arXiv:2611.03725 — arXiv (en), https://arxiv.org/abs/2611.03725 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f04064299618 (Metraipolis archive dc_f04064299618, sha256 439b2d1b0209f33dadd48eb702ca0279b0a8ae530d9c61526ebbe900dfa2b667)
BibTeX
@misc{dcf04064299618,
  title = {From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines — arXiv:2611.03725},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03725},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f04064299618; sha256 439b2d1b0209f33dadd48eb702ca0279b0a8ae530d9c61526ebbe900dfa2b667},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03725. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Retrieval-Augmented Generation (RAG) pipelines are widely used for domain-specific question answering, yet locating faults remains manual and time-consuming. Fault localization is challenging because RAG pipelines consist of interdependent stages, where faults can propagate downstream, masking their origin, or remain undetected even when the final answer is correct. Existing evaluation approaches often focus on individual stages and rely on reference answers or code instrumentation. We present RAGLog, which detects and localizes faults across RAG stages using execution logs, without requiring reference answers or code instrumentation. RAGLog transforms heterogeneous API logs into a common RAG schema and combines rule-based, embedding-based, and LLM-based checks to identify faulty stages. We introduce a live fault injector and RAGLog Faults, a dataset of 3,134 runs from five RAG pipelines with stage-level fault labels. With Claude Haiku 5.5, RAGLog achieves a pooled F1 score of 0.73 on runs with naturally occurring faults, outperforming Claude Code (0.58) and Codex (0.52), and comparable effectiveness on runs with injected faults (0.84 versus 0.80 and 0.82, respectively). It analyzes runs in approximately five seconds at US\$0.005 each, with overall lower cost and latency than agent baselines. RAGLog is available as an open-source tool for debugging, evaluating, and monitoring RAG pipelines.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.03725 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:21:13.490Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).