Metraipolis › archive › dc_636933dfbae2
AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems — arXiv:2609.37864
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.37864
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:27:49.651Z
- SHA-256 of the archived text
bcd5eef6fb1f502193a4b10539aed3a3e5ea1ca16db3bb6dcb43035154c56620- Stable URL of this page
https://metraipolis.com/documento/dc_636933dfbae2
How to cite
AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems — arXiv:2609.37864 — arXiv (en), https://arxiv.org/abs/2609.37864 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_636933dfbae2 (Metraipolis archive dc_636933dfbae2, sha256 bcd5eef6fb1f502193a4b10539aed3a3e5ea1ca16db3bb6dcb43035154c56620)
BibTeX
@misc{dc636933dfbae2,
title = {AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems — arXiv:2609.37864},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.37864},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_636933dfbae2; sha256 bcd5eef6fb1f502193a4b10539aed3a3e5ea1ca16db3bb6dcb43035154c56620},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37864. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Agent harness bugs exhibit unique characteristics and remain challenging for state-of-the-art software agents to repair. Progress in this area is further hindered by existing benchmarks, which contain only a small and fixed number of executable harness bugs while requiring hundreds of human hours to construct. This work presents AgentBug-Smith, an automated harness bug reproduction approach that continuously discovers and reproduces real-world harness bugs from open-source agentic systems. Across different backbone LLMs, AgentBug-Smith consistently outperforms existing bug reproduction techniques designed for general software, achieving 10.67% - 27.56% higher success rates of reproducing harness bugs. By applying AgentBug-Smith to open-source agentic systems in the wild, we construct Live-Harness-Bench, a live and extensible benchmark that currently contains 200 reproducible harness bugs. We further demonstrate the utility of Live-Harness-Bench through two downstream applications. First, we use Live-Harness-Bench as the evaluation benchmark to systematically evaluate state-of-the-art software agents, revealing their limited capabilities in repairing real-world harness bugs. Second, we use Live-Harness-Bench as a knowledge base of real-world harness bug fixes, from which reusable repair skills can be distilled to improve existing software agents, increasing their harness-bug repair rates by 6.32%. Together, AgentBug-Smith and Live-Harness-Bench establish a scalable foundation for continuously evaluating and improving software agents on harness bug repair, turning real-world agent failures into executable evaluation instances and reusable knowledge for harness improvement, thus contributing to the ultimate goal of recursively self-improving agents. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.37864 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T15:54:54.040Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).