Metraipolis › archive › dc_06c2bbb839fa
How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices — arXiv:2610.11877
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11877
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T05:48:28.694Z
- SHA-256 of the archived text
ac35761ebfa3c2e2bbc4dd70389193dba1b2b41cf9cdbd58ce0f8fdbc0bc6cf0- Stable URL of this page
https://metraipolis.com/documento/dc_06c2bbb839fa
How to cite
How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices — arXiv:2610.11877 — arXiv (en), https://arxiv.org/abs/2610.11877 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_06c2bbb839fa (Metraipolis archive dc_06c2bbb839fa, sha256 ac35761ebfa3c2e2bbc4dd70389193dba1b2b41cf9cdbd58ce0f8fdbc0bc6cf0)
BibTeX
@misc{dc06c2bbb839fa,
title = {How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices — arXiv:2610.11877},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11877},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_06c2bbb839fa; sha256 ac35761ebfa3c2e2bbc4dd70389193dba1b2b41cf9cdbd58ce0f8fdbc0bc6cf0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11877. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Automated research systems support literature synthesis, ideation, experiments, writing, and peer review, but their evaluation is dispersed across tasks, benchmarks, and studies that are difficult to compare directly. We review this literature from the perspective of evaluation design and evidence, covering six targets: literature synthesis, research ideation, executable workflows, scholarly writing and communication, automatic peer review, and end-to-end research. We compare task construction, evidence sources, evaluators, and scoring procedures to explain the capabilities assessed by different designs. Our synthesis highlights three recurring lessons: output checks, process checks, and human studies provide complementary information; evaluator calibration is specific to the property being assessed; and resource budgets and attempt selection are integral to interpreting performance comparisons. We identify diagnostic evaluation designs and documented gaps in supporting evidence, and translate these comparisons into reporting and audit recommendations for specific evaluation settings. The survey helps readers navigate existing evaluations, select appropriate benchmarks, and design subsequent studies. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.11877 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:00.990Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).