Metraipolis › archive › dc_fb03604d61b7
MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses — arXiv:2609.33411
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33411
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:56:46.202Z
- SHA-256 of the archived text
d4b4f883c988fe29268877e164d91410f0fd28a9c9f6e80941b4e4ae87a12bcb- Stable URL of this page
https://metraipolis.com/documento/dc_fb03604d61b7
How to cite
MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses — arXiv:2609.33411 — arXiv (en), https://arxiv.org/abs/2609.33411 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb03604d61b7 (Metraipolis archive dc_fb03604d61b7, sha256 d4b4f883c988fe29268877e164d91410f0fd28a9c9f6e80941b4e4ae87a12bcb)
BibTeX
@misc{dcfb03604d61b7,
title = {MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses — arXiv:2609.33411},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33411},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb03604d61b7; sha256 d4b4f883c988fe29268877e164d91410f0fd28a9c9f6e80941b4e4ae87a12bcb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33411. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Rapid progress in Large Language Models (LLMs) is saturating static benchmarks faster than they can be designed. While existing automated evolution frameworks attempt to generate harder questions by perturbing individual tasks, they remain constrained by rigid, hard-coded generation rules. Moving beyond the evolution of isolated tasks, we propose to optimize the benchmark generation workflow itself end to end with MetaBench-Harness, a dual-loop search framework. Specifically, the inner loop utilizes a benchmark harness to generate a new benchmark in each round, while the outer meta-harness orchestration layer iteratively refines and searches over harness implementations based on historical evolution trajectories. By applying MetaBench-Harness to the competitive programming CodeContests and Olympiad mathematics AIME-2024 datasets, we demonstrate that the evolved benchmarks are challenging and discriminative for frontier models. Trajectory and quality analyses verify that MetaBench-Harness enables multi-dimensional evolution, steadily improving evolution reasonableness, benchmark competency, and evaluator robustness across successive rounds. Furthermore, case studies reveal its effective utilization of diverse difficulty levers to reframe problems and elevate required capabilities. Ultimately, this work provides a solution to the pressing challenge of benchmark saturation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2609.33411 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.246Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).