Metraipolis › archive › dc_f6fe00cedbbd

Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model — arXiv:2608.13867

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.13867
License (as archived): CC0-1.0
Archived: 2026-10-19T02:21:10.402Z
SHA-256 of the archived text: c92bfc2ca02f392cd850876b74f5982aca80ce1e435e2d0f0228001e094ff33c
Stable URL of this page: https://metraipolis.com/documento/dc_f6fe00cedbbd

How to cite

Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model — arXiv:2608.13867 — arXiv (en), https://arxiv.org/abs/2608.13867 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f6fe00cedbbd (Metraipolis archive dc_f6fe00cedbbd, sha256 c92bfc2ca02f392cd850876b74f5982aca80ce1e435e2d0f0228001e094ff33c)

BibTeX

@misc{dcf6fe00cedbbd,
  title = {Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model — arXiv:2608.13867},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13867},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f6fe00cedbbd; sha256 c92bfc2ca02f392cd850876b74f5982aca80ce1e435e2d0f0228001e094ff33c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13867. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

AI coding agents are commonly evaluated as models but deployed as systems. Their reliability depends not only on model capability, but on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation. This monograph examines those boundaries and develops a framework for evaluating and operating coding agents reliably. It synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records through a structured multivocal review, targeted update audits, software-engineering coverage analysis, and distributed-systems evidence synthesis. Across this evidence, many apparent model failures originate elsewhere in the system, while improvements at one layer often fail to propagate to end-to-end outcomes. Evaluation and operation are treated as a dependency chain in which weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate downstream conclusions. The monograph contributes a versioned catalog of 206 reliability records: 193 gated practices, including 56 developed in depth, plus 13 research leads; an evidence ledger; a framework for dependency and repair asymmetry across the agent lifecycle; measurements and failure cases from operated agent systems; runnable evaluation and reliability protocols; and five reusable agent skills with evidence maps. Together, these provide a system-level methodology for distinguishing model capability from infrastructure effects, designing defensible evaluations, and building systems that recover safely when components fail. The review is structured rather than exhaustive, evidence strength varies by topic, and results depend on workload and configuration. The methods record which search lanes were executed, which remain unexecuted, and limits on evidence-grading claims.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.13867 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.877Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).