Metraipolis › archive › dc_3b698f367fbc
Do Coding Agents Catch Their Own Bugs? (They Don't.) — arXiv:2611.01181
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01181
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:27:16.829Z
- SHA-256 of the archived text
2e3012109861a9899b932477ab18ab223cee8c2a25beda81682cecd27d6c41c2- Stable URL of this page
https://metraipolis.com/documento/dc_3b698f367fbc
How to cite
Do Coding Agents Catch Their Own Bugs? (They Don't.) — arXiv:2611.01181 — arXiv (en), https://arxiv.org/abs/2611.01181 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3b698f367fbc (Metraipolis archive dc_3b698f367fbc, sha256 2e3012109861a9899b932477ab18ab223cee8c2a25beda81682cecd27d6c41c2)
BibTeX
@misc{dc3b698f367fbc,
title = {Do Coding Agents Catch Their Own Bugs? (They Don't.) — arXiv:2611.01181},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01181},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3b698f367fbc; sha256 2e3012109861a9899b932477ab18ab223cee8c2a25beda81682cecd27d6c41c2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01181. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In software development, engineers need to foresee and fix potential problems with their code that are not explicitly specified in requirements---such as pre-emptively considering vulnerabilities or other bugs from a code change. But current benchmarks often do not measure these future-looking consequences. We introduce $\textbf{Foresight-SWE}$, a benchmark for measuring whether coding agents can anticipate and address unstated problems during software engineering tasks. Foresight-SWE consists of 212 samples spanning 4 languages and 35 popular open-source repositories. Each sample contains two pull requests (PRs), the second of which resolves defects the first one introduced. To measure whether agents exhibit foresight, we define the $\textbf{foresight resolve rate (FRR)}$ as the fraction of instances where an agent fully resolves the original task while also avoiding the defect identified by a later PR. We evaluate eleven large language models and find that FRR ranges from only 6.13% to 24.53%, indicating that models largely fail to address problems introduced by the first PR change. This raises important questions about how models are trained and aligned in coding contexts, as well as how code review tools should be improved for more fail-safe coding. Foresight-SWE shifts evaluations in this direction.
Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.01181 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:52.185Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).