Metraipolis › archive › dc_edd47cd27441

LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems — arXiv:2609.37143

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.37143
License (as archived): CC0-1.0
Archived: 2026-10-07T16:57:37.789Z
SHA-256 of the archived text: d9f151adf4cfcebbcd2ea772da2622f97b508ee0d1a9b36350ce5e43610ce007
Stable URL of this page: https://metraipolis.com/documento/dc_edd47cd27441

How to cite

LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems — arXiv:2609.37143 — arXiv (en), https://arxiv.org/abs/2609.37143 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_edd47cd27441 (Metraipolis archive dc_edd47cd27441, sha256 d9f151adf4cfcebbcd2ea772da2622f97b508ee0d1a9b36350ce5e43610ce007)

BibTeX

@misc{dcedd47cd27441,
  title = {LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems — arXiv:2609.37143},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37143},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_edd47cd27441; sha256 d9f151adf4cfcebbcd2ea772da2622f97b508ee0d1a9b36350ce5e43610ce007},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37143. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks with large reference implementations. Many benchmarks evaluate coding agents' implementation capability to produce correct code edits from detailed specifications. However, practical modular development tasks also require the perception capability of grounding user intent and high-level design to derive a specification. We introduce LoLBench to evaluate both capabilities through the entire proposal-to-implementation process on large software systems. It is a multilingual benchmark of 100 tasks across 29 software systems in five domains. Each task provides a human-written enhancement proposal with user intent and high-level design. On average, proposals contain about 5,000 words, software systems contain 2.4 million source lines of code (LoC), and implementation pull requests (PRs) change approximately 5,500 LoC. Across 28 agents we evaluated, the best agent resolves only 14% of tasks and achieves a 52.7% Fail-to-Pass (F2P) pass rate. Failure analysis identifies incomplete code localization as a major bottleneck, while providing reference-derived file trees alongside API specifications improves resolved rates by 16--22 percentage points (2.4--17$\times$), reaching at most 34%. These results show that both perception and implementation remain central challenges for coding agents in practical modular development on large software systems. LoLBench is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL).
Categorie arXiv: Software Engineering (cs.SE) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.37143 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:27:03.440Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).