Metraipolis › archive › dc_74b055490346
Construct Validity of Coding Benchmarks for LLM-based Agentic Coding Tools — arXiv:2611.05665
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.05665
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-05T18:26:23.561Z
- SHA-256 of the archived text
1ce2080fa652b1b2ce6018d3654ffbd7238f699ee25fdc4c63199a86feafb11d- Stable URL of this page
https://metraipolis.com/documento/dc_74b055490346
How to cite
Construct Validity of Coding Benchmarks for LLM-based Agentic Coding Tools — arXiv:2611.05665 — arXiv (en), https://arxiv.org/abs/2611.05665 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_74b055490346 (Metraipolis archive dc_74b055490346, sha256 1ce2080fa652b1b2ce6018d3654ffbd7238f699ee25fdc4c63199a86feafb11d)
BibTeX
@misc{dc74b055490346,
title = {Construct Validity of Coding Benchmarks for LLM-based Agentic Coding Tools — arXiv:2611.05665},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.05665},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_74b055490346; sha256 1ce2080fa652b1b2ce6018d3654ffbd7238f699ee25fdc4c63199a86feafb11d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05665. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Software developers increasingly adopt LLM-based agentic coding tools to assist them with development tasks. To provide developers with up-to-date information on these tools' effectiveness, prior work introduced benchmarks that enable comparisons across tools. Despite the widespread adoption of such benchmarks, however, recent work has shown that they suffer from issues threatening their validity. Aims. In this work, we aim to complement prior work by analyzing the construct validity of agentic coding benchmarks. Methodology. We analyze nine benchmarks commonly used to evaluate the effectiveness of agentic coding tools. We identify the constructs that the benchmarks claim to measure, the metrics used to operationalize them, and potential confounders that interfere with the measurement. Results. Our findings show that benchmarks commonly operationalize broad concepts such as coding capability with only a small set of metrics. The reported attributes that might confound the measurements vary between benchmarks, indicating a lack of standardization. Conclusions. We conclude with recommendations to strengthen construct validity in agentic coding benchmarks and encourage further empirical work to better understand the effectiveness of agentic coding tools. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2611.05665 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-05T18:16:03.879Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).