Metraipolis › archive › dc_5cc2710ac9bf
Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes — arXiv:2610.11963
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11963
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:03:23.520Z
- SHA-256 of the archived text
9a23437121c51f6c27cc63a156c30d9fdfd1f437944612a4b9a0b8334205be86- Stable URL of this page
https://metraipolis.com/documento/dc_5cc2710ac9bf
How to cite
Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes — arXiv:2610.11963 — arXiv (en), https://arxiv.org/abs/2610.11963 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5cc2710ac9bf (Metraipolis archive dc_5cc2710ac9bf, sha256 9a23437121c51f6c27cc63a156c30d9fdfd1f437944612a4b9a0b8334205be86)
BibTeX
@misc{dc5cc2710ac9bf,
title = {Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes — arXiv:2610.11963},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11963},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5cc2710ac9bf; sha256 9a23437121c51f6c27cc63a156c30d9fdfd1f437944612a4b9a0b8334205be86},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11963. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A no-code fix resolves an invalid bug report by directing the user to change a setting, update to a version where the problem is already fixed, or adjust their workflow. Manually verifying whether a proposed no-code fix resolves the reported bug takes considerable developer time. This study proposes an automated, execution-based pipeline for evaluating the capability of large language models (LLMs) to generate no-code fixes in a real browser environment. We evaluate 322 no-code fixes generated by the 12 configurations released with the benchmark of a previous study, covering bug reports categorized as Faulty Configuration, Wrong Version, or External System & Dependency. An executor agent applies each fix by following its natural-language instructions, and an issue-specific checker determines whether the reported bug persists. We repeat the pipeline with three executors: two Computer-Use Agents, OpenCUA-72B and Claude Sonnet 5, and one multimodal agentic LLM, Meta's Muse Glimmer. Only 17.6% of the candidate issues could be set up and passed both sanity gates. Across the 322 fixes, 14.6% to 49.7% resolved the bug depending on the executor, and the strongest configuration, Claude Opus 4.6 in the Vanilla pipeline, resolved up to 74.1% of its fixes under Claude Sonnet 5. Changing only the executor shifted a configuration's resolution rate by 38.8% on average, and the three executors reached the same verdict on only 46.9% of the fixes. Compared with human execution, the executors matched the human consensus for 66.1% to 88.1% of the sampled fixes. Even under the best executor, fewer than half of the LLM-generated no-code fixes resolve the reported bug, so such fixes need verification before they reach users. Execution-based verification can provide this, but the measured capability depends strongly on the executor, which evaluations must report and control. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.11963 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:01.988Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).