Metraipolis › archive › dc_554106c98087
AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents — arXiv:2610.05140
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.05140
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:24:59.931Z
- SHA-256 of the archived text
15258f5cc0aa9c6bfc02aa7ed5aea8c85f162a64be815b72bbe3379c54bd0f5d- Stable URL of this page
https://metraipolis.com/documento/dc_554106c98087
How to cite
AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents — arXiv:2610.05140 — arXiv, https://arxiv.org/abs/2610.05140 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_554106c98087 (Metraipolis archive dc_554106c98087, sha256 15258f5cc0aa9c6bfc02aa7ed5aea8c85f162a64be815b72bbe3379c54bd0f5d)
BibTeX
@misc{dc554106c98087,
title = {AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents — arXiv:2610.05140},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.05140},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_554106c98087; sha256 15258f5cc0aa9c6bfc02aa7ed5aea8c85f162a64be815b72bbe3379c54bd0f5d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05140. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluation aligned with advances in agent capabilities. We address this challenge by investigating whether scientific-agent benchmarks can be automatically generated and iteratively adapted as agent capabilities evolve. We introduce AutoSciBench, a framework that represents each task as a high-level concept specifying the scientific domain, data modality, and required reasoning approach, together with a low-level recipe specifying how the question, environment, and ground-truth answer are constructed and verified. Agents attempt to solve each task, producing solver trajectories and corresponding judge feedback which AutoSciBench uses to revise the recipe or concept, closing observed shortcuts and shifting tasks toward raw-data re-examination, interpretation of intermediate results, and evidence integration. Experience distilled from completed refinement trajectories further guides new concept generation, allowing lessons from earlier task refinement to inform subsequent benchmark construction. Starting from existing benchmarks, we evaluate AutoSciBench across computational biology, materials science, and clinical imaging. Generated benchmarks reduce average solver accuracy by 22.4 and 25.5 percentage points relative to the human-curated benchmarks in computational biology and materials science, respectively, while generated tasks receive higher average quality ratings across all three domains, suggesting that scientific-agent evaluation can adapt as agent capabilities advance. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.05140 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:05.552Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).