Metraipolis › archive › dc_f67ea34971ed

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead — arXiv:2609.17394

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.17394
License (as archived): CC0-1.0
Archived: 2026-10-15T18:38:56.337Z
SHA-256 of the archived text: 0d23f250c9fe86d83d1d5bee74cc65ab4a944bb5bbaa7be64bf80b88ebdbe4e9
Stable URL of this page: https://metraipolis.com/documento/dc_f67ea34971ed

How to cite

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead — arXiv:2609.17394 — arXiv (en), https://arxiv.org/abs/2609.17394 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f67ea34971ed (Metraipolis archive dc_f67ea34971ed, sha256 0d23f250c9fe86d83d1d5bee74cc65ab4a944bb5bbaa7be64bf80b88ebdbe4e9)

BibTeX

@misc{dcf67ea34971ed,
  title = {Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead — arXiv:2609.17394},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.17394},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f67ea34971ed; sha256 0d23f250c9fe86d83d1d5bee74cc65ab4a944bb5bbaa7be64bf80b88ebdbe4e9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.17394. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Frontier solution sets have median nesting 0.935 against a score-implied baseline of 0.774, indicating strongly shared successes. Scores also depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction; non-rejection does not establish equivalence. We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution. The results motivate reporting comparison-set-specific resolution and model-scaffold provenance instead of interpreting small aggregate gaps as established rank differences.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.17394 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:03:09.749Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).