Metraipolis › archive › dc_58df954698e1

Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection — arXiv:2609.37669

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.37669
License (as archived): CC0-1.0
Archived: 2026-10-07T16:27:33.761Z
SHA-256 of the archived text: 23dc5c0822170bae25e0e88e832f0182b9aa990b2eefa8e37f88433bce23cf91
Stable URL of this page: https://metraipolis.com/documento/dc_58df954698e1

How to cite

Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection — arXiv:2609.37669 — arXiv (en), https://arxiv.org/abs/2609.37669 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_58df954698e1 (Metraipolis archive dc_58df954698e1, sha256 23dc5c0822170bae25e0e88e832f0182b9aa990b2eefa8e37f88433bce23cf91)

BibTeX

@misc{dc58df954698e1,
  title = {Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection — arXiv:2609.37669},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37669},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_58df954698e1; sha256 23dc5c0822170bae25e0e88e832f0182b9aa990b2eefa8e37f88433bce23cf91},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37669. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Retrieval-Augmented Generation (RAG) is increasingly used to enhance Large Language Model (LLM)-based software vulnerability detection by grounding predictions in retrieved vulnerability knowledge, such as vulnerability reports. However, existing RAG-based software vulnerability detection (RAG4SVD) systems are often evaluated using proprietary models, which challenges open science and reproducibility. Further, studies use different datasets, custom knowledge bases, different backbone models, and diverse metrics, which hinders meaningful cross-system comparison. In this work, we study six open-source RAG4SVD systems and address these reproducibility and comparability challenges through (i) reproduction of their experimental settings under an open-weight setting, and (ii) a unified benchmark using a common dataset, metric suite, and pool of open-weight models. Further, RAG4SVD systems typically consist of multiple components, yet are often evaluated only as a whole system, i.e., end-to-end. Therefore, we perform (iii) a component-level analysis that decomposes representative RAG4SVD pipelines into input abstraction, knowledge retrieval, and detection. Our results demonstrate that reproducibility varies substantially across systems. Under the presented unified benchmark, published RAG4SVD performance does not transfer under a controlled open-weight evaluation and depends strongly on the used model. The component analysis shows that effective RAG4SVD depends on the alignment between pipeline stages. For example, oracle knowledge raises retrieval to near-optimal, yet performance remains low (0.51 pairwise accuracy), demonstrating that retrieval effectiveness alone is insufficient for reliable detection. These findings motivate evaluating RAG4SVD not only end-to-end, but at the level of pipeline components, and provide a basis for more standardized, RAG-aware evaluation practices.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.37669 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:54.012Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).