Metraipolis › archive › dc_b3a12cf85bbe
On the Reliability of LLM-Based Vulnerability Patching Benchmarks — arXiv:2610.10150
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.10150
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:44:32.450Z
- SHA-256 of the archived text
5610c38287f2fdd7c379461f58a22b56fad6a984b90f53d2ad8c5437a65d7e02- Stable URL of this page
https://metraipolis.com/documento/dc_b3a12cf85bbe
How to cite
On the Reliability of LLM-Based Vulnerability Patching Benchmarks — arXiv:2610.10150 — arXiv (en), https://arxiv.org/abs/2610.10150 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b3a12cf85bbe (Metraipolis archive dc_b3a12cf85bbe, sha256 5610c38287f2fdd7c379461f58a22b56fad6a984b90f53d2ad8c5437a65d7e02)
BibTeX
@misc{dcb3a12cf85bbe,
title = {On the Reliability of LLM-Based Vulnerability Patching Benchmarks — arXiv:2610.10150},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.10150},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b3a12cf85bbe; sha256 5610c38287f2fdd7c379461f58a22b56fad6a984b90f53d2ad8c5437a65d7e02},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10150. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) have shown strong potential for automated vulnerability patching, but current benchmarks can substantially distort reported performance. Drawing on extensive experience developing, running, and stress-testing such frameworks, we identify under-examined pitfalls across three dimensions: (1) agent-level factors, where prompting, tool availability, and detailed instructions can raise success rates without improving developer-aligned patch quality; (2) framework-level factors, where permission errors, infrastructure bugs, and timeout handling can silently suppress or inflate performance; and (3) dataset-level factors, where bug reports and single proof-of-concept (PoC) tests fail to capture whether patches address root causes or follow developer intent. We curate 112 historical bugs from 84 open-source C/C++, Go, and Rust projects, each with PoC tests, regression tests, and additional developer tests that assess alignment with the original developers' design principles. Through controlled experiments and case studies, we show that LLMs can achieve high PoC passing rates under ideal conditions, yet benchmark execution choices can materially change measured success. More importantly, developer-test passing rates remain low and improve only marginally with newer models, suggesting that models increasingly suppress symptoms without consistently producing upstream-quality fixes. These results show that benchmark scores are highly sensitive to evaluation design, and we provide practical guidelines for more rigorous, reliable, and reproducible evaluation. Tema degli esploratori: fiducia digitale e crittografia (elenco arXiv cs.CR, nuovi lavori del giorno). Categorie arXiv: Cryptography and Security (cs.CR) ; Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2610.10150 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:45:22.462Z dalla pagina https://arxiv.org/list/cs.CR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).