Metraipolis › archive › dc_ed82afbda3e4
A Study of the Reliability of Agentic AI-Generated Programs — arXiv:2609.18298
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.18298
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T15:38:48.254Z
- SHA-256 of the archived text
c9737c6037a93af6737486127954502516122bb65beef0026061551888701d53- Stable URL of this page
https://metraipolis.com/documento/dc_ed82afbda3e4
How to cite
A Study of the Reliability of Agentic AI-Generated Programs — arXiv:2609.18298 — arXiv (en), https://arxiv.org/abs/2609.18298 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ed82afbda3e4 (Metraipolis archive dc_ed82afbda3e4, sha256 c9737c6037a93af6737486127954502516122bb65beef0026061551888701d53)
BibTeX
@misc{dced82afbda3e4,
title = {A Study of the Reliability of Agentic AI-Generated Programs — arXiv:2609.18298},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.18298},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ed82afbda3e4; sha256 c9737c6037a93af6737486127954502516122bb65beef0026061551888701d53},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18298. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Agentic-AI based software development offers the promise of faster completion of the software, greater programmer efficiency, and more reliable code. The question is how can we verify these claims in an objective way? In this project, we attempted to answer this question based on three practices. First, we applied a typical best-practices agentic AI workflow for software development. Second, our target programs were ten well-known, release-quality human-written Linux utility programs so that we could compare the AI-generated code against a concrete ground truth. Third, we based our measure of reliability on a widely used testing technique, fuzz random testing. For this testing, we used both classic black box, generational testing and more modern coverage guided (gray box, mutational) testing using AFL++. We found that the AI-generated versions of the utility programs were typically as reliable - often more reliable - than the latest human-generated versions of these programs. While the AI-generated versions did have some failures, they were less common than the code from the standard repositories. Interestingly, the AI-generated code was less likely to have failures such as memory errors (such as buffer overflows) but more likely to have hangs such as infinite loops. In addition, we verified that generating robust and reliable software using agentic AI requires careful practice and human supervision. The quality of the code is highly dependent on the prompts and skills used, and how the human directing the process responds. We also demonstrated that using agentic AI workflow for software development (with its prompts and skills) can become a specification of the code that leads to cost-effective sustainability of the software. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-17 in cs.SE). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.18298 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T15:04:30.921Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).