Metraipolis › archive › dc_0cbc43d65602
EvalBuilder: Can LLMs Build Efficient Benchmarks? — arXiv:2610.20099
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20099
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T20:02:21.080Z
- SHA-256 of the archived text
7244dd31d95763882fb88859e19bccde40419e0265c0acfc891d07ae857a10be- Stable URL of this page
https://metraipolis.com/documento/dc_0cbc43d65602
How to cite
EvalBuilder: Can LLMs Build Efficient Benchmarks? — arXiv:2610.20099 — arXiv (en), https://arxiv.org/abs/2610.20099 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0cbc43d65602 (Metraipolis archive dc_0cbc43d65602, sha256 7244dd31d95763882fb88859e19bccde40419e0265c0acfc891d07ae857a10be)
BibTeX
@misc{dc0cbc43d65602,
title = {EvalBuilder: Can LLMs Build Efficient Benchmarks? — arXiv:2610.20099},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20099},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0cbc43d65602; sha256 7244dd31d95763882fb88859e19bccde40419e0265c0acfc891d07ae857a10be},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20099. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. Based on this formulation, we introduce EvalBuilder, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.20099 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T19:35:48.289Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).