Metraipolis › archive › dc_7c9e4c8b0db2
ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval — arXiv:2609.01865
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.01865
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:11:39.755Z
- SHA-256 of the archived text
6b945accc06d5a72438899d6a78709ebe799113a6b9308b4d082853c9b3159e2- Stable URL of this page
https://metraipolis.com/documento/dc_7c9e4c8b0db2
How to cite
ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval — arXiv:2609.01865 — arXiv (en), https://arxiv.org/abs/2609.01865 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7c9e4c8b0db2 (Metraipolis archive dc_7c9e4c8b0db2, sha256 6b945accc06d5a72438899d6a78709ebe799113a6b9308b4d082853c9b3159e2)
BibTeX
@misc{dc7c9e4c8b0db2,
title = {ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval — arXiv:2609.01865},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.01865},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7c9e4c8b0db2; sha256 6b945accc06d5a72438899d6a78709ebe799113a6b9308b4d082853c9b3159e2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01865. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Retrieval (cs.IR). Pagina del lavoro: https://arxiv.org/abs/2609.01865 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T00:28:02.925Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).