Metraipolis › archive › dc_cd4d6816ffe3
Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair — arXiv:2609.04909
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04909
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T22:28:46.651Z
- SHA-256 of the archived text
1a6c0c07293a60c0db45c01ef5f4398a423f6cf3fff97ec58eb763137fcdc7d5- Stable URL of this page
https://metraipolis.com/documento/dc_cd4d6816ffe3
How to cite
Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair — arXiv:2609.04909 — arXiv (en), https://arxiv.org/abs/2609.04909 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd4d6816ffe3 (Metraipolis archive dc_cd4d6816ffe3, sha256 1a6c0c07293a60c0db45c01ef5f4398a423f6cf3fff97ec58eb763137fcdc7d5)
BibTeX
@misc{dccd4d6816ffe3,
title = {Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair — arXiv:2609.04909},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04909},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd4d6816ffe3; sha256 1a6c0c07293a60c0db45c01ef5f4398a423f6cf3fff97ec58eb763137fcdc7d5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04909. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation. We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.04909 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T22:18:11.553Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).