Metraipolis › archive › dc_e3aa36fc5ce7
VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities — arXiv:2609.08040
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.08040
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:40:27.542Z
- SHA-256 of the archived text
72dc56daf00e5e1a5fcab9ea3adaf54aaa3b39831e8b74968f4ee6d7a1d16373- Stable URL of this page
https://metraipolis.com/documento/dc_e3aa36fc5ce7
How to cite
VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities — arXiv:2609.08040 — arXiv (en), https://arxiv.org/abs/2609.08040 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e3aa36fc5ce7 (Metraipolis archive dc_e3aa36fc5ce7, sha256 72dc56daf00e5e1a5fcab9ea3adaf54aaa3b39831e8b74968f4ee6d7a1d16373)
BibTeX
@misc{dce3aa36fc5ce7,
title = {VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities — arXiv:2609.08040},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.08040},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e3aa36fc5ce7; sha256 72dc56daf00e5e1a5fcab9ea3adaf54aaa3b39831e8b74968f4ee6d7a1d16373},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08040. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing defenses such as GitHub Dependabot often raise many false alerts because their coarse-grained matching cannot determine whether a vulnerable dependency is actually exploitable. Security analysts typically spend substantial time assessing vulnerability exploitability case by case. Recent LLM agents have emerged as promising candidates for this task given their advanced capabilities in coding and cybersecurity, yet no existing benchmark evaluates them on it. Prior benchmarks target zero-day settings, where agents detect and exploit previously unknown vulnerabilities. In contrast, software supply chain security focuses on how known vulnerabilities in upstream dependencies affect downstream projects. This requires agents to reason across repositories and determine whether an upstream vulnerability is exploitable in the downstream project. To address this gap, we introduce VEX-Bench, the first benchmark for evaluating LLM agents' ability to assess the exploitability of software supply chain vulnerabilities. It contains 75 real-world cases mined from GitHub and labeled by security experts, covering Python, Java, and Go. We evaluate nine models across three agent harnesses. While GPT-5.5 and Claude Opus 4.6 reach approximately 80% F1 on binary vulnerability-status classification, only GPT-5.5 surpasses 70% macro-F1 on fine-grained justification classification. This gap highlights the challenge of moving beyond binary exploitability assessment to identifying fine-grained exploitability reasons. Code and data: this https URL Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CR). Categorie arXiv: Cryptography and Security (cs.CR) ; Machine Learning (cs.LG); Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2609.08040 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:09:48.492Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).