Metraipolis › archive › dc_f29873c2d403
SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs? — arXiv:2609.21190
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.21190
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T01:07:51.353Z
- SHA-256 of the archived text
7b2aa062b4232d04de0e4baa7d82812d54e4e3f80ed2f608c62590416203368e- Stable URL of this page
https://metraipolis.com/documento/dc_f29873c2d403
How to cite
SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs? — arXiv:2609.21190 — arXiv (en), https://arxiv.org/abs/2609.21190 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f29873c2d403 (Metraipolis archive dc_f29873c2d403, sha256 7b2aa062b4232d04de0e4baa7d82812d54e4e3f80ed2f608c62590416203368e)
BibTeX
@misc{dcf29873c2d403,
title = {SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs? — arXiv:2609.21190},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.21190},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f29873c2d403; sha256 7b2aa062b4232d04de0e4baa7d82812d54e4e3f80ed2f608c62590416203368e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21190. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check correctness with held-out test suites, which are inherently incomplete and increasingly susceptible to memorization. Formal verification avoids both problems, but existing work covers only standalone tasks whose specifications are given as input, not real issues, which touch large repositories and state intent in vague natural language. We present Benchproofer, a pipeline that turns a coding task with a known correct patch into a formally verified one: it writes a specification for the new code, summarizes the existing functions that code calls with axioms, and admits an instance only after mechanical and adversarial gates agree. Applying it to SWE-bench Verified yields SWE-Proof, 500 real issues whose correctness is formally verified rather than tested, and it extends to SWE-bench Pro. Evaluating Claude Opus 4.8, we find that verification catches what tests miss: a quarter of test-passing patches admit counterexamples, which a structured natural-language specification does not fix, while a correct formal one lifts resolution from 85% to 95%. Writing that specification is the hard part: an agent that must write its own gains nothing over an unaided baseline, and only 56% of those specifications pass our audit. The usual failure is faithfulness, a specification that constrains part of the required behavior and leaves the rest free. Specification quality still tracks the outcome, failing on 92% of unresolved instances against 51% of resolved ones, making faithful specification synthesis a concrete open problem. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2609.21190 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-10T00:23:42.512Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).