Metraipolis › archive › dc_003b03036404
When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems — arXiv:2609.10548
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.10548
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T05:46:48.882Z
- SHA-256 of the archived text
5487158d3c5e31d15e3b6f53dbce58be83ebfdeaa1c664f23aefc856bfa461df- Stable URL of this page
https://metraipolis.com/documento/dc_003b03036404
How to cite
When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems — arXiv:2609.10548 — arXiv (en), https://arxiv.org/abs/2609.10548 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_003b03036404 (Metraipolis archive dc_003b03036404, sha256 5487158d3c5e31d15e3b6f53dbce58be83ebfdeaa1c664f23aefc856bfa461df)
BibTeX
@misc{dc003b03036404,
title = {When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems — arXiv:2609.10548},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.10548},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_003b03036404; sha256 5487158d3c5e31d15e3b6f53dbce58be83ebfdeaa1c664f23aefc856bfa461df},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.10548. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM-based agents for automated code repair have received significant attention in recent years from both research and software engineering practice perspectives. However, limited attention has been paid to patches that pass syntactic and functional verification but still retain or introduce security vulnerabilities. The aim of this research is to systematically identify and categorize such silent failures in LLM-based agentic code repair. We conducted an empirical study using 1,030 valid execution traces produced by seven agent frameworks with GPT-4o-mini across two security-focused datasets, SecurityEval and CVEfixes. Through three iterations of qualitative coding and manual verification, 170 confirmed silent failures were identified. The key results are: (i) Three main categories of silent failures were identified: Omission, Introduction, and Inadequacy. Omission accounts for 48.2% of the confirmed failures, Introduction for 30.6%, and Inadequacy for 21.2%. (ii) Ten fine-grained failure codes were classified under these three categories, showing how agents omit required security controls, apply incomplete defenses, or introduce new vulnerabilities during repair. (iii) Current test-passing evaluation and LLM-based reviewer roles were insufficient to expose or intercept these failures in the confirmed cases. (iv) Similar insecure solutions appeared across different frameworks, suggesting possible shared model-, prompt-, or task-level influences, while single-agent and multi-agent systems showed different failure profiles. The results of this study will assist researchers and practitioners in improving the evaluation of LLM-based agentic code repair and developing targeted verification methods that go beyond functional correctness and cover all generated artifacts. Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-09-11 in cs.CR). Categorie arXiv: Software Engineering (cs.SE) ; Cryptography and Security (cs.CR). Pagina del lavoro: https://arxiv.org/abs/2609.10548 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T05:30:39.409Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-09-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).