Metraipolis › archive › dc_0df8a63e3e89

TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution — arXiv:2610.10619

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.10619
License (as archived)
CC0-1.0
Archived
2026-10-09T05:53:33.855Z
SHA-256 of the archived text
93665fe40b93c903e3d59dcd4bc0f07a159f6a17f137d5dcffcec20559b1f216
Stable URL of this page
https://metraipolis.com/documento/dc_0df8a63e3e89

How to cite

TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution — arXiv:2610.10619 — arXiv (en), https://arxiv.org/abs/2610.10619 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0df8a63e3e89 (Metraipolis archive dc_0df8a63e3e89, sha256 93665fe40b93c903e3d59dcd4bc0f07a159f6a17f137d5dcffcec20559b1f216)
BibTeX
@misc{dc0df8a63e3e89,
  title = {TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution — arXiv:2610.10619},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.10619},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0df8a63e3e89; sha256 93665fe40b93c903e3d59dcd4bc0f07a159f6a17f137d5dcffcec20559b1f216},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10619. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so agents can reward hack them or silently miss required behavior while still passing every test. As a result, higher benchmark scores may partly reflect better adaptation to the evaluator rather than better problem solving. Existing works focus on static test augmentation: they strengthen each task's tests once, before any trial is seen, and thus overlook how real trials actually fail. We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests. For each trial, TestJack generates tests targeting prompt requirements the patch may violate, retains only tests passed by the ground-truth patch, and re-examines any trial failures. Each confirmed failure is thus supported by a replayable test. To reduce evaluation cost, we also introduce a lightweight variant which audits a random sample of trials in depth and reuses the resulting tests across all trials for the same task. Across 6 frontier model backends and 5 benchmarks such as DeepSWE and SWE Marathon, we find that about 34.4% of the model trials currently judged correct violate the task requirements, lowering the overall resolution rate from 50.6% to 33.2%. Our results reveal a fundamental limitation of current coding-agent evaluation: as LLMs become better at optimizing against fixed evaluators, those evaluators themselves must become more adaptive.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.10619 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:01.210Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).