Metraipolis › archive › dc_57f2cd69d366

PatchBench: Evaluating AI Agents for Vulnerability Patching — arXiv:2609.04075

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.04075
License (as archived): CC0-1.0
Archived: 2026-10-17T01:12:17.066Z
SHA-256 of the archived text: ae1d8f281fea3db6086a420e9d4c66b9913c196f4ece1799f2f7d235c98bb310
Stable URL of this page: https://metraipolis.com/documento/dc_57f2cd69d366

How to cite

PatchBench: Evaluating AI Agents for Vulnerability Patching — arXiv:2609.04075 — arXiv (en), https://arxiv.org/abs/2609.04075 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_57f2cd69d366 (Metraipolis archive dc_57f2cd69d366, sha256 ae1d8f281fea3db6086a420e9d4c66b9913c196f4ece1799f2f7d235c98bb310)

BibTeX

@misc{dc57f2cd69d366,
  title = {PatchBench: Evaluating AI Agents for Vulnerability Patching — arXiv:2609.04075},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04075},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_57f2cd69d366; sha256 ae1d8f281fea3db6086a420e9d4c66b9913c196f4ece1799f2f7d235c98bb310},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04075. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

AI agents have recently demonstrated strong performance in automated vulnerability patching. However, existing evaluations often validate a patch only by testing whether the provided Proof-of-Concept (PoC) input still triggers a crash. This leaves two key threats to validity: agents may reproduce memorized historical developer patches, or they may generate surface-level fixes that only suppress the reported crash. We study these concerns for C/C++ vulnerability patching. We introduce a patch similarity metric to detect memorized patches. On average, 25% of the agent patches exhibit substantial similarity to historical developer patches, indicating that patch memorization is a real threat to the validity of vulnerability patching evaluations. Meanwhile, agents also frequently exploit benchmark structures to pass patch validation by patching on the crash stack trace to suppress the crash, rather than localizing and fixing the root cause of the vulnerabilities. To handle these issues, we propose PatchBench, a new benchmark for evaluating AI agents on realistic vulnerability patching tasks. PatchBench selects vulnerabilities whose ground-truth fixes lie outside the crash stack and uses vulnerability transplant and code mutations to migrate historical vulnerabilities into new repository contexts, reducing the risks of surface-level fixes and patch memorization. We develop new patch validation methods that thoroughly evaluate both security and semantic correctness of agent patches. Across 11 state-of-the-art agents, including the top three AIxCC agents, the original PoC-only validation inflates the patching task solve rate of agents by 1.83$\times$ on average. Our results reveal key limitations of current patching agents and point to future research directions for more reliable vulnerability repair.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.04075 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T00:28:03.030Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).