Metraipolis › archive › dc_ce06aefda060

EvalResearchBench: Can AI Agents Design Their Own Evaluations? — arXiv:2610.04184

Source
arXiv
Original
https://arxiv.org/abs/2610.04184
License (as archived)
CC0-1.0
Archived
2026-10-06T05:24:21.502Z
SHA-256 of the archived text
e34e1802339859f57438c5cb592c08ef3022c0f262e1880b475d0a47722dbd3c
Stable URL of this page
https://metraipolis.com/documento/dc_ce06aefda060

How to cite

EvalResearchBench: Can AI Agents Design Their Own Evaluations? — arXiv:2610.04184 — arXiv, https://arxiv.org/abs/2610.04184 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ce06aefda060 (Metraipolis archive dc_ce06aefda060, sha256 e34e1802339859f57438c5cb592c08ef3022c0f262e1880b475d0a47722dbd3c)
BibTeX
@misc{dcce06aefda060,
  title = {EvalResearchBench: Can AI Agents Design Their Own Evaluations? — arXiv:2610.04184},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04184},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ce06aefda060; sha256 e34e1802339859f57438c5cb592c08ef3022c0f262e1880b475d0a47722dbd3c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04184. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75\% of candidate pairs as the targets do, below the 91\% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.04184 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.436Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).