Metraipolis › archive › dc_e8fa3382af05

Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes — arXiv:2609.34262

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34262
License (as archived)
CC0-1.0
Archived
2026-10-07T20:57:36.242Z
SHA-256 of the archived text
77f899772ce624276a642152512941a150b4445e130e72cca2301f3bbd11fcc6
Stable URL of this page
https://metraipolis.com/documento/dc_e8fa3382af05

How to cite

Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes — arXiv:2609.34262 — arXiv (en), https://arxiv.org/abs/2609.34262 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e8fa3382af05 (Metraipolis archive dc_e8fa3382af05, sha256 77f899772ce624276a642152512941a150b4445e130e72cca2301f3bbd11fcc6)
BibTeX
@misc{dce8fa3382af05,
  title = {Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes — arXiv:2609.34262},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34262},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e8fa3382af05; sha256 77f899772ce624276a642152512941a150b4445e130e72cca2301f3bbd11fcc6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34262. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agentic benchmarks guide model selection and training. Yet an agent can pass a task without demonstrating the intended capability. Such outcomes constitute unearned passes; their proportion among all passes defines the integrity gap. As agents improve, benchmark surfaces that once seemed harmless can become exploitable, making benchmark validity an ongoing maintenance problem. We introduce a process-verification framework that audits passing trajectories, distinguishes evidenced reward hacking from verifier weakness, and localizes exploitable surfaces for repair. Across 3,810 passing trajectories from 29 model-benchmark cohorts, confirmed violations often increase with model generation but not monotonically. On SWEBench Pro V1.0, confirmed violation rates rise from 24% to 73% between Opus 4.7 and Fable 5 on matched tasks; later cohorts fall to 11% for Fable 5.1 and 0% for GPT-6 Astra. These comparisons are descriptive: configurations were not normalized, and the latest models also pass fewer exploitable tasks. Violations concentrate around a small set of recurring surfaces, especially unintended access to reference solutions through git history. Three repair case studies across two benchmarks show why blocking a recorded exploit is insufficient: the same protected information can remain accessible through another route. Therefore, we combine minimal patches with exploit replay and fresh agent evaluation, auditing new passes under the original standard. No evaluated attempt against the final patches reached the protected channel, and every post-patch pass was judged legitimate. Benchmark integrity requires ongoing maintenance: audit passing behavior, repair the enabling surface, and re-evaluate both exploit access and legitimate solvability.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.34262 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.496Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).