Metraipolis › archive › dc_de4bc67e784a

Harvesting SWE Benchmarks from Wild Vibe Coding Sessions — arXiv:2611.02440

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02440
License (as archived): CC0-1.0
Archived: 2026-11-03T11:07:41.180Z
SHA-256 of the archived text: 005d17eaf3faea86e9890840e661a343ce259f24708bd0f5d40a567c7dfa0424
Stable URL of this page: https://metraipolis.com/documento/dc_de4bc67e784a

How to cite

Harvesting SWE Benchmarks from Wild Vibe Coding Sessions — arXiv:2611.02440 — arXiv (en), https://arxiv.org/abs/2611.02440 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_de4bc67e784a (Metraipolis archive dc_de4bc67e784a, sha256 005d17eaf3faea86e9890840e661a343ce259f24708bd0f5d40a567c7dfa0424)

BibTeX

@misc{dcde4bc67e784a,
  title = {Harvesting SWE Benchmarks from Wild Vibe Coding Sessions — arXiv:2611.02440},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02440},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_de4bc67e784a; sha256 005d17eaf3faea86e9890840e661a343ce259f24708bd0f5d40a567c7dfa0424},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02440. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Development teams need reproducible tasks from their own repositories and workflows to evaluate coding agents. Conversational developer-agent collaboration ("vibe coding") produces abundant coding sessions and repository checkpoints, but requirements evolve across turns, unrelated conversations co-occur with commits, and engineering efforts can span multiple sessions. We present SessionFarmer, an automated framework that turns these records into self-contained, executable software engineering tasks. It analyzes dialogue and commit history to filter unrelated sessions and group multi-turn, potentially multi-session interactions into work episodes centered on one engineering goal. For each episode, it reconstructs a hermetic execution container, synthesizes a task statement from the full context, and authors repository-grounded engineering-quality rubrics and unit tests. Closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptances of incomplete solutions. We construct CropChat, an open benchmark of 242 tasks across 61 public repositories and 14 programming languages, and CropEnterprise from proprietary enterprise logs. In a CropEnterprise audit, the original developers endorse the vast majority of reconstructed episode boundaries, task statements, and quality rubrics. Full-episode context covers substantially more user requirements than single-turn or single-session inputs. Test refinement cuts false-rejection rates and raises balanced accuracy against reference patch labels by 7-9 percentage points across solvers. Across 12 model-harness configurations on CropChat, the strongest setup achieves 39.3% Pass@1, while harness choice shifts Pass@1 by up to 8.4 percentage points for the same model.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02440 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:46.099Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).