Metraipolis › archive › dc_09e813611478
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks — arXiv:2608.14905
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14905
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T18:56:38.488Z
- SHA-256 of the archived text
9f60385044df3a0f18b9004b0373660cc6b73fa73b1222703948664a54d41c7c- Stable URL of this page
https://metraipolis.com/documento/dc_09e813611478
How to cite
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks — arXiv:2608.14905 — arXiv (en), https://arxiv.org/abs/2608.14905 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_09e813611478 (Metraipolis archive dc_09e813611478, sha256 9f60385044df3a0f18b9004b0373660cc6b73fa73b1222703948664a54d41c7c)
BibTeX
@misc{dc09e813611478,
title = {How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks — arXiv:2608.14905},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14905},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_09e813611478; sha256 9f60385044df3a0f18b9004b0373660cc6b73fa73b1222703948664a54d41c7c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14905. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry whole-stage research from an initial hypothesis all the way to final published paper, which is a paradigm now referred to as AutoResearch. Existing evaluations reveal little about how these agents operate or where they break down. Tasks are narrowly-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility. To address this gap, we introduce AutoResearchEval, featuring 100 tasks grounded in published frontier science across 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories, with process-level annotation. We organize these insights into AutoResearch Failure Taxonomy or ARFT, a framework of 45 empirically-grounded failure patterns. To enable scalable fine-grained attribution, we leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts. Failure patterns converge on a single overarching limitation, namely that current agents lack a metacognitive loop, which entails the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound. The same patterns recur across all 8 harness-model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration-level interventions can close it is an open question this work does not test. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.14905 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T18:06:57.836Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).