Metraipolis › archive › dc_a7252d082c46

Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent — arXiv:2610.39417

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.39417
License (as archived)
CC0-1.0
Archived
2026-10-31T06:13:15.836Z
SHA-256 of the archived text
67aa951a54beaacffa735aecff385243ed1edeafc39bac1d7d8d5709b0ce12ff
Stable URL of this page
https://metraipolis.com/documento/dc_a7252d082c46

How to cite

Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent — arXiv:2610.39417 — arXiv (en), https://arxiv.org/abs/2610.39417 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a7252d082c46 (Metraipolis archive dc_a7252d082c46, sha256 67aa951a54beaacffa735aecff385243ed1edeafc39bac1d7d8d5709b0ce12ff)
BibTeX
@misc{dca7252d082c46,
  title = {Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent — arXiv:2610.39417},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.39417},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a7252d082c46; sha256 67aa951a54beaacffa735aecff385243ed1edeafc39bac1d7d8d5709b0ce12ff},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.39417. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) have shown strong performance on software engineering (SE) tasks, driving the need for high-quality datasets for reliable training and evaluation. SWE-bench-style datasets (SWE datasets) are widely used to evaluate LLM-based coding agents on real-world software issues, where task clarity and test coverage directly affect whether instances are solvable and verifiable. However, quality issues persist in existing SWE datasets, and manual annotation is costly and labor-intensive, motivating LLM-based annotation. Existing approaches typically rely on static, overly general rubrics, limiting their ability to handle diverse and nuanced scenarios and often yielding high false positive rates. In contrast, human annotators draw on accumulated experience to make context-aware judgments. Inspired by this, we propose ExLabel, an LLM-based, self-evolving, experience-guided assistant for task clarity and test coverage assessment in SWE datasets. ExLabel maintains an evolving experience bank that distills cross-instance knowledge into stable yet adaptive judgment criteria and reuses relevant experience to guide the assessment of new instances. For systematic evaluation, we construct SWE-TQ, a new benchmark spanning eight programming languages, by cleaning human annotations from SWE-bench and Multi-SWE-bench. Experiments with three backbone LLMs show that ExLabel achieves the best F1 across all settings, with relative improvements of 44.2%/49.9% and 24.0%/18.4% over the best baseline for task clarity and test coverage assessment on the Python/Multilingual subsets, respectively. Moreover, a coding model fine-tuned on SWE-rebench data filtered by ExLabel resolves 66.2% of issues on SWE-bench Verified, outperforming the model trained on data filtered by the best baseline by 14.5%.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.39417 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:08.717Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).