Metraipolis › archive › dc_c4c28a8634bc

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests — arXiv:2608.27831

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27831
License (as archived)
CC0-1.0
Archived
2026-10-17T06:52:26.945Z
SHA-256 of the archived text
6813c4f7466f8da578d56a1d37eb77d1a61c0908c175e4a628c0f7d5e4429cf3
Stable URL of this page
https://metraipolis.com/documento/dc_c4c28a8634bc

How to cite

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests — arXiv:2608.27831 — arXiv (en), https://arxiv.org/abs/2608.27831 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c4c28a8634bc (Metraipolis archive dc_c4c28a8634bc, sha256 6813c4f7466f8da578d56a1d37eb77d1a61c0908c175e4a628c0f7d5e4429cf3)
BibTeX
@misc{dcc4c28a8634bc,
  title = {RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests — arXiv:2608.27831},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27831},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c4c28a8634bc; sha256 6813c4f7466f8da578d56a1d37eb77d1a61c0908c175e4a628c0f7d5e4429cf3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27831. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.LG).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.27831 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:41:55.870Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).