Metraipolis › archive › dc_1f6c65bbfe3d
Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents — arXiv:2610.32578
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.32578
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-26T20:15:42.397Z
- SHA-256 of the archived text
ce1e6fa9e75251eb814e82286dab965bcb930832fb1a177db33c2c22256dc049- Stable URL of this page
https://metraipolis.com/documento/dc_1f6c65bbfe3d
How to cite
Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents — arXiv:2610.32578 — arXiv (en), https://arxiv.org/abs/2610.32578 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1f6c65bbfe3d (Metraipolis archive dc_1f6c65bbfe3d, sha256 ce1e6fa9e75251eb814e82286dab965bcb930832fb1a177db33c2c22256dc049)
BibTeX
@misc{dc1f6c65bbfe3d,
title = {Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents — arXiv:2610.32578},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.32578},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1f6c65bbfe3d; sha256 ce1e6fa9e75251eb814e82286dab965bcb930832fb1a177db33c2c22256dc049},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32578. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
AI coding agents increasingly work inside a harness: rule files such as this http URL , on-demand skills, runtime hooks and automated checks. Most studies of harnesses look at a single point in time or at short benchmark runs. I report a six-week case study of my personal system of record, a plain-markdown repository that three AI agent tools edited daily under one shared rule set. I built a ledger of 118 agent failures, their fixes and whether they came back. I also tested the system's monthly LLM documentation audit against 47 known stale statements and 20 planted errors. Finally, I replayed five past failures in isolated git clones, with and without their fixes. All results come from one repository with three runs per condition, so I report them as exploratory. Stale documentation (drift) was the largest failure class (48 of 118), although 35 of those were found on one audit day. A single audit run found about half of the known drift (Claude Sonnet 5.5 mean: 50%), and at least 93% of its findings were correct. On Sonnet 5.5, the agent finished four of the five replayed tasks. In three of those four, the failure did not happen, with or without its fix. Only the stale-count failure came back without its check, and the check stopped it in every run. On older models, an agent drafted an em-dash once while the em-dash lint check was active, and the check rejected it. A written rule on run-time estimates did not change Sonnet 4.5's overestimates. These results suggest that harness lessons can outlive the failures they were written to fix, while drift persists and still needs mechanical checks. Tools that evolve a harness add lessons faster than they retire them, so they should re-test their rules from time to time and retire the obsolete ones. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2610.32578 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-26T19:49:18.379Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).