Metraipolis › archive › dc_42a17cbfd2d5

Harvesting SWE Benchmarks from Wild Vibe Coding Sessions — arXiv:2611.02440

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02440
License (as archived)
CC0-1.0
Archived
2026-11-03T18:19:54.414Z
SHA-256 of the archived text
1a841ae2d8d16e7526f634c61914153b30d5ebfcb13ead43d110458f4219d238
Stable URL of this page
https://metraipolis.com/documento/dc_42a17cbfd2d5

How to cite

Harvesting SWE Benchmarks from Wild Vibe Coding Sessions — arXiv:2611.02440 — arXiv (en), https://arxiv.org/abs/2611.02440 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_42a17cbfd2d5 (Metraipolis archive dc_42a17cbfd2d5, sha256 1a841ae2d8d16e7526f634c61914153b30d5ebfcb13ead43d110458f4219d238)
BibTeX
@misc{dc42a17cbfd2d5,
  title = {Harvesting SWE Benchmarks from Wild Vibe Coding Sessions — arXiv:2611.02440},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02440},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_42a17cbfd2d5; sha256 1a841ae2d8d16e7526f634c61914153b30d5ebfcb13ead43d110458f4219d238},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02440. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Development teams need reproducible tasks from their own repositories and workflows to evaluate coding agents. Conversational developer-agent collaboration ("vibe coding") produces abundant coding sessions and repository checkpoints, but requirements evolve across turns, unrelated conversations co-occur with commits, and engineering efforts can span multiple sessions. We present SessionFarmer, an automated framework that turns these records into self-contained, executable software engineering tasks. It analyzes dialogue and commit history to filter unrelated sessions and group multi-turn, potentially multi-session interactions into work episodes centered on one engineering goal. For each episode, it reconstructs a hermetic execution container, synthesizes a task statement from the full context, and authors repository-grounded engineering-quality rubrics and unit tests. Closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptances of incomplete solutions. We construct CropChat, an open benchmark of 242 tasks across 61 public repositories and 14 programming languages, and CropEnterprise from proprietary enterprise logs. In a CropEnterprise audit, the original developers endorse the vast majority of reconstructed episode boundaries, task statements, and quality rubrics. Full-episode context covers substantially more user requirements than single-turn or single-session inputs. Test refinement cuts false-rejection rates and raises balanced accuracy against reference patch labels by 7-9 percentage points across solvers. Across 12 model-harness configurations on CropChat, the strongest setup achieves 39.3% Pass@1, while harness choice shifts Pass@1 by up to 8.4 percentage points for the same model.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.02440 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.681Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).