Metraipolis › archive › dc_ef315db8af93
SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction — arXiv:2609.06229
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.06229
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T14:09:02.213Z
- SHA-256 of the archived text
20ec30f010107873c4b74d0de677d38c55609a5fcb00993af49dc3df319fbc9f- Stable URL of this page
https://metraipolis.com/documento/dc_ef315db8af93
How to cite
SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction — arXiv:2609.06229 — arXiv (en), https://arxiv.org/abs/2609.06229 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef315db8af93 (Metraipolis archive dc_ef315db8af93, sha256 20ec30f010107873c4b74d0de677d38c55609a5fcb00993af49dc3df319fbc9f)
BibTeX
@misc{dcef315db8af93,
title = {SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction — arXiv:2609.06229},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.06229},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef315db8af93; sha256 20ec30f010107873c4b74d0de677d38c55609a5fcb00993af49dc3df319fbc9f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06229. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Vulnerability discovery is becoming an important ability of large language model (LLM) agents: agents that silently miss real defects leave critical software exposed. Rigorously measuring this ability is therefore urgent, but existing benchmarks are gameable through data contamination, score recall against an unknowable vulnerability set, often rely on synthetic bugs, and report a single end-to-end verdict that cannot localize where an agent fails. Vulnerability discovery is a composite ability: an agent must comprehend source code, infer input constraints, construct inputs, execute them, and iteratively correct from feedback. We recast its measurement as an input-prediction task with a closed, deterministic ground truth: using coverage-guided fuzzing, we mine deep target branches in real-world C/C++ programs and ask an agent to predict an input that drives execution to a given branch. This decomposes discovery into three task modes over 22 real-world C/C++ programs spanning 15 domains. Open-loop and Feedback-enabled share 60 fixed-target task instances across 16 of these codebases (13 domains), testing input construction without and with a distance oracle to isolate code comprehension from feedback-driven correction. Online Arena instead removes the predefined target and scores path exploration by coverage gain on a separate, partially overlapping pool of 11 programs; agents collectively confirmed 13 distinct bugs across six programs. Evaluating 15 default-effort model-scaffold configurations, the best reaches only 55.0% pass rate in the Feedback-enabled mode, and the mean across seven paired Claude Code configurations is 36.4% with feedback versus 19.3% without. Decomposing failures, we find constraint inference, not navigation, is the dominant bottleneck. We release SWE-Test with a turnkey evaluation environment. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-09 in cs.SE). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.06229 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:58:36.364Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).