Metraipolis › archive › dc_787accb5e67e
Take It From Here: Evaluating Coding Agents On Project Takeover — arXiv:2611.07307
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.07307
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:41:25.696Z
- SHA-256 of the archived text
cf6d1c3d12e27240bc54a29de34d014c44e2b766db33bd67a40f073c0d74ce58- Stable URL of this page
https://metraipolis.com/documento/dc_787accb5e67e
How to cite
Take It From Here: Evaluating Coding Agents On Project Takeover — arXiv:2611.07307 — arXiv (en), https://arxiv.org/abs/2611.07307 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_787accb5e67e (Metraipolis archive dc_787accb5e67e, sha256 cf6d1c3d12e27240bc54a29de34d014c44e2b766db33bd67a40f073c0d74ce58)
BibTeX
@misc{dc787accb5e67e,
title = {Take It From Here: Evaluating Coding Agents On Project Takeover — arXiv:2611.07307},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.07307},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_787accb5e67e; sha256 cf6d1c3d12e27240bc54a29de34d014c44e2b766db33bd67a40f073c0d74ce58},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07307. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Coding agents increasingly collaborate on long-term projects. When taking over a project, they need to recover currently applicable requirements from the interaction history and update them in response to new requests. However, most coding benchmarks do not include long interaction histories, and some evaluations that consider history still do not cover the full process from requirement reconstruction to project delivery. We therefore provide agents with the interaction history, current request, and pre-request project environment, ask them to make the requested changes, and assess takeover performance through delivery tests. Across six model-harness configurations and 37 targets eligible for executable validation, the mean complete-delivery rate under Full History is only 4.95%. We then use requirement atoms to independently diagnose a larger set of takeover targets, examining relevant information identification, pre-request state reconstruction, and current request processing. We further supply correct pre-request or target states to examine the remaining difficulties in requirement processing and project modification. The diagnostics show that none of the six configurations produces fully correct post-request requirement states, and a case illustrates omission of previously recovered constraints during updating. With correct target states supplied, the mean complete-delivery rate is 8.11%, with build and regression pass rates of 96.40% and 90.99%, but target-test success remains 9.01%. These results show difficulties in both forming target requirement states and implementing supplied targets in existing projects. Based on these findings, we introduce ReqMemBench, which combines requirement diagnostics with delivery validation to support systematic evaluation of project takeover. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.MA, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE) ; Multiagent Systems (cs.MA). Pagina del lavoro: https://arxiv.org/abs/2611.07307 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-06T14:18:27.667Z dalla pagina https://arxiv.org/list/cs.MA/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).