Metraipolis › archive › dc_48755565b930

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch — arXiv:2609.38335

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.38335
License (as archived): CC0-1.0
Archived: 2026-10-07T12:43:31.440Z
SHA-256 of the archived text: 96ded791c1508a589807faea6e73c6e1cf5f81393b61cc91cbc8b0891fa94b24
Stable URL of this page: https://metraipolis.com/documento/dc_48755565b930

How to cite

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch — arXiv:2609.38335 — arXiv (en), https://arxiv.org/abs/2609.38335 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_48755565b930 (Metraipolis archive dc_48755565b930, sha256 96ded791c1508a589807faea6e73c6e1cf5f81393b61cc91cbc8b0891fa94b24)

BibTeX

@misc{dc48755565b930,
  title = {E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch — arXiv:2609.38335},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38335},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_48755565b930; sha256 96ded791c1508a589807faea6e73c6e1cf5f81393b61cc91cbc8b0891fa94b24},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38335. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.38335 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.856Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).