Metraipolis › archive › dc_0cbc43d65602

EvalBuilder: Can LLMs Build Efficient Benchmarks? — arXiv:2610.20099

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20099
License (as archived)
CC0-1.0
Archived
2026-10-15T20:02:21.080Z
SHA-256 of the archived text
7244dd31d95763882fb88859e19bccde40419e0265c0acfc891d07ae857a10be
Stable URL of this page
https://metraipolis.com/documento/dc_0cbc43d65602

How to cite

EvalBuilder: Can LLMs Build Efficient Benchmarks? — arXiv:2610.20099 — arXiv (en), https://arxiv.org/abs/2610.20099 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0cbc43d65602 (Metraipolis archive dc_0cbc43d65602, sha256 7244dd31d95763882fb88859e19bccde40419e0265c0acfc891d07ae857a10be)
BibTeX
@misc{dc0cbc43d65602,
  title = {EvalBuilder: Can LLMs Build Efficient Benchmarks? — arXiv:2610.20099},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20099},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0cbc43d65602; sha256 7244dd31d95763882fb88859e19bccde40419e0265c0acfc891d07ae857a10be},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20099. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. Based on this formulation, we introduce EvalBuilder, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.20099 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T19:35:48.289Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).