Metraipolis › archive › dc_2a11875f84ef
DeltaSelect: Affordable A/B Testing for Coding Agents — arXiv:2609.19607
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.19607
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T03:23:24.640Z
- SHA-256 of the archived text
54b8c64d0f6df66f15c05580c0d077768b53e2b2c8301cc3b80d2e0c0cc1cd11- Stable URL of this page
https://metraipolis.com/documento/dc_2a11875f84ef
How to cite
DeltaSelect: Affordable A/B Testing for Coding Agents — arXiv:2609.19607 — arXiv (en), https://arxiv.org/abs/2609.19607 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2a11875f84ef (Metraipolis archive dc_2a11875f84ef, sha256 54b8c64d0f6df66f15c05580c0d077768b53e2b2c8301cc3b80d2e0c0cc1cd11)
BibTeX
@misc{dc2a11875f84ef,
title = {DeltaSelect: Affordable A/B Testing for Coding Agents — arXiv:2609.19607},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.19607},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2a11875f84ef; sha256 54b8c64d0f6df66f15c05580c0d077768b53e2b2c8301cc3b80d2e0c0cc1cd11},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19607. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Coding-agent benchmarks are built for broad and comprehensive comparisons, not frequent development decisions. Individual runs vary, full suites are expensive, and the benchmark harness may differ from the harness used in practice. In a resampling analysis of DeepSWE's published trials, only 19.5% of tasks (22 of 113) had a fifth-percentile Pearson correlation of at least 0.50 with full-benchmark performance. The paper presents DeltaSelect, an open-source method that identifies tasks whose one-run results consistently track full-benchmark performance using Pearson correlation, maps fractional verifier results to a common score using linear regression, and selects a fixed task set within a dollar budget. DeltaSelect is intended for repeated baseline-versus-candidate comparisons during development, not model rankings. In a gpt-5.6-luna low-reasoning case study, DeltaSelect was used to revise custom skills and instructions. Across 13 evaluations, the recorded cost was USD 27.86 at rates published August 16, 2026. The adopted version cost 58.1% less than the initial version (USD 1.75 versus USD 4.18; p=0.008), while the calibrated score was higher (42.36% versus 36.46%; published-analog variance p=0.326). Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.19607 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-10T03:04:13.838Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).