Metraipolis › archive › dc_6e60a08eb093
Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation — arXiv:2610.05830
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.05830
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:25:22.219Z
- SHA-256 of the archived text
9e6802d963ccc5fe12c5bc81cfe5a7059e21a80daf032bc7388d156ec30e61ae- Stable URL of this page
https://metraipolis.com/documento/dc_6e60a08eb093
How to cite
Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation — arXiv:2610.05830 — arXiv, https://arxiv.org/abs/2610.05830 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6e60a08eb093 (Metraipolis archive dc_6e60a08eb093, sha256 9e6802d963ccc5fe12c5bc81cfe5a7059e21a80daf032bc7388d156ec30e61ae)
BibTeX
@misc{dc6e60a08eb093,
title = {Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation — arXiv:2610.05830},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.05830},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6e60a08eb093; sha256 9e6802d963ccc5fe12c5bc81cfe5a7059e21a80daf032bc7388d156ec30e61ae},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05830. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Agentic leaderboards increasingly evaluate systems on public benchmarks whose task statements and solution-bearing artifacts can remain accessible. We propose a measurement-first audit framework that distinguishes contamination claims according to the evidence required to support them. It separates three channels that require different evidence: training-time exposure, evaluation-time retrieval, and pipeline/scaffold leakage. Each channel is coded as open, partial, closed, or unknown under a fail-closed rule. Across nine Holistic Agent Leaderboard (HAL) configurations, none of the 27 channel assessments was coded closed, but incidents were confirmed in four configurations. We then apply the behavioral component of the framework to a reported file-localization gap on SWE-bench Verified, using an outcome-blind, same-repository matched-control design with symmetric prompt-leakage screening, paired and repository-aware uncertainty analyses, and scorer validation, evaluated on GPT-4.1 and DeepSeek-V4-Flash. Among the 100 pairs retained after symmetric screening and the pair-integrity exclusion, GPT-4.1 showed a $+10.0$-point pair-weighted Top-3 benchmark-associated gap, but the 95\% intervals from both the prespecified paired-bootstrap procedure and the post-hoc repository-balanced analysis included zero, leaving the benchmark-associated gap inconclusive. The reproduction scorer did not pass its validation gate: against consensus human labels, sufficient scorer sensitivity could not be established for either model, and both DeepSeek-V4-Flash firings on correct-gold comparisons were false positives. Without provenance evidence, appropriate controls, symmetric leakage screening, and validated scorers, stronger contamination claims are not warranted. The results do not establish training-data membership, contamination prevalence, or benchmark-induced score inflation. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.05830 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:05.636Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).