Metraipolis › archive › dc_95b4e84d8eea
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents — arXiv:2609.08149
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.08149
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T14:09:15.128Z
- SHA-256 of the archived text
0c9c088bc1d1a050fb8e6a47df840833aa67fffe9839bb4cbfa1b6b658f77453- Stable URL of this page
https://metraipolis.com/documento/dc_95b4e84d8eea
How to cite
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents — arXiv:2609.08149 — arXiv (en), https://arxiv.org/abs/2609.08149 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_95b4e84d8eea (Metraipolis archive dc_95b4e84d8eea, sha256 0c9c088bc1d1a050fb8e6a47df840833aa67fffe9839bb4cbfa1b6b658f77453)
BibTeX
@misc{dc95b4e84d8eea,
title = {SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents — arXiv:2609.08149},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.08149},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_95b4e84d8eea; sha256 0c9c088bc1d1a050fb8e6a47df840833aa67fffe9839bb4cbfa1b6b658f77453},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08149. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously evaluated, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-09 in cs.SE). Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2609.08149 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:58:36.431Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).