Metraipolis › archive › dc_651ee00ef0d8

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer — arXiv:2609.31587

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.31587
License (as archived)
CC0-1.0
Archived
2026-10-08T01:40:01.693Z
SHA-256 of the archived text
7bb9c8f1400e028fe7ce9dca9ba519a3d04475c6b3ca7e0f05184845b9530efc
Stable URL of this page
https://metraipolis.com/documento/dc_651ee00ef0d8

How to cite

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer — arXiv:2609.31587 — arXiv (en), https://arxiv.org/abs/2609.31587 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_651ee00ef0d8 (Metraipolis archive dc_651ee00ef0d8, sha256 7bb9c8f1400e028fe7ce9dca9ba519a3d04475c6b3ca7e0f05184845b9530efc)
BibTeX
@misc{dc651ee00ef0d8,
  title = {Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer — arXiv:2609.31587},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31587},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_651ee00ef0d8; sha256 7bb9c8f1400e028fe7ce9dca9ba519a3d04475c6b3ca7e0f05184845b9530efc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31587. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the benchmark as an optimization signal, we discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the benchmark and the optimizer, and we characterize the boundary at which documentation helps.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.31587 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:12:14.617Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).