Metraipolis › archive › dc_aeaecb9787b4

When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor — arXiv:2609.01985

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01985
License (as archived)
CC0-1.0
Archived
2026-10-17T03:24:16.019Z
SHA-256 of the archived text
1a82e22c70d3488a0981680412e3757c6b7f3cf394f6e5e8e20267193cf4ce31
Stable URL of this page
https://metraipolis.com/documento/dc_aeaecb9787b4

How to cite

When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor — arXiv:2609.01985 — arXiv (en), https://arxiv.org/abs/2609.01985 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aeaecb9787b4 (Metraipolis archive dc_aeaecb9787b4, sha256 1a82e22c70d3488a0981680412e3757c6b7f3cf394f6e5e8e20267193cf4ce31)
BibTeX
@misc{dcaeaecb9787b4,
  title = {When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor — arXiv:2609.01985},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01985},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aeaecb9787b4; sha256 1a82e22c70d3488a0981680412e3757c6b7f3cf394f6e5e8e20267193cf4ce31},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01985. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As LLM coding agents increasingly perform end-to-end engineering work, we lack empirical characterization of how they behave on systems-level requirements: schema design, async orchestration, configuration correctness, and retrieval-filtering trade-offs. We present a case study of one such agent implementing a multi-component data system against a detailed pre-existing specification. Storage technologies, schema, entity-resolution algorithm, and retrieval-filtering strategy were fixed in advance; the agent autonomy was in the implementation, in diagnosing and fixing defects it introduced, and in interaction-design choices left open. Over a single session, we catalog five such defects, categorized by constraint violated and detection method. We further evaluate, on the public HotpotQA benchmark, the one retrieval trade-off specified in that architecture: restricting candidates to a graph-identified entity set before ranking versus unfiltered search. We substitute the benchmark gold evidence labels for entity identification, since we lacked LLM access to run that stage, and report standard recall rather than the benchmark own accuracy metrics. Across retrieval budgets from 1 to 10 and 100 questions against a pooled corpus of 2994 paragraphs, filtered recall reaches its ceiling by a budget of 3, expected once candidates are restricted to the gold paragraphs themselves, while unfiltered search recovers all required evidence only 69 percent of the time even at a budget of 10, a gap that holds at every budget tested, with sign test p less than 0.0001. We close with a discussion of where the agent autonomy succeeded versus required correction, including one instance where a claimed performance fix was never re-measured on the regression that motivated it.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.MA).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2609.01985 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T03:12:51.777Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).