Metraipolis › archive › dc_4c5fca613e10

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks — arXiv:2609.18805

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18805
License (as archived)
CC0-1.0
Archived
2026-10-15T15:38:53.783Z
SHA-256 of the archived text
01afebbf5a5346dcf430ee01056b12b8819c31dc549b9cb87a4066b34898b4a0
Stable URL of this page
https://metraipolis.com/documento/dc_4c5fca613e10

How to cite

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks — arXiv:2609.18805 — arXiv (en), https://arxiv.org/abs/2609.18805 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4c5fca613e10 (Metraipolis archive dc_4c5fca613e10, sha256 01afebbf5a5346dcf430ee01056b12b8819c31dc549b9cb87a4066b34898b4a0)
BibTeX
@misc{dc4c5fca613e10,
  title = {ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks — arXiv:2609.18805},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18805},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4c5fca613e10; sha256 01afebbf5a5346dcf430ee01056b12b8819c31dc549b9cb87a4066b34898b4a0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18805. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-17 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.18805 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T15:04:30.925Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).