Metraipolis › archive › dc_e66ca9fe3b59
The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents — arXiv:2608.28795
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.28795
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:47:15.761Z
- SHA-256 of the archived text
1b77680908e2d269a809c3df515c7ecd019f31b4641cc8af4f8805ef015272a7- Stable URL of this page
https://metraipolis.com/documento/dc_e66ca9fe3b59
How to cite
The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents — arXiv:2608.28795 — arXiv (en), https://arxiv.org/abs/2608.28795 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e66ca9fe3b59 (Metraipolis archive dc_e66ca9fe3b59, sha256 1b77680908e2d269a809c3df515c7ecd019f31b4641cc8af4f8805ef015272a7)
BibTeX
@misc{dce66ca9fe3b59,
title = {The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents — arXiv:2608.28795},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.28795},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e66ca9fe3b59; sha256 1b77680908e2d269a809c3df515c7ecd019f31b4641cc8af4f8805ef015272a7},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28795. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screenshot tool. We call this set the agent's verification surface. This study asks whether increasing only that surface, with everything else held fixed, produces a matching growth in the quality of the software the agent ships. We built a minimal coding agent whose tool list is the single controlled variable and used it to implement 1,116 web applications across six models and eight tool configurations. A condition-blind human graded every application against a frozen rubric, and automatic probes stress-tested the API-observable behaviors. Verification's cheapest benefit arrives first, which is to make sure that the application comes up. Without any tools, about one build in seven fails to launch at all and a single boot probe removes nearly all of these failures at roughly 35 percent of a full shell's token cost, while the full shell multiplies the no-tools cost by 2.35. Screenshots help most where mistakes are visible (e.g. element placement, interaction), though even there the gain over a shell is modest and does not survive correction for multiple statistical comparisons. In cases where failures can only be measured rather than seen, such as keeping scrolling smooth over a 100,000-row list, screenshots add nothing. A verification tool improves the output artifact only where its reach covers the way the application actually fails. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.28795 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.683Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).