Metraipolis › archive › dc_e35e92e50d99
AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment — arXiv:2610.33644
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.33644
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T07:57:11.538Z
- SHA-256 of the archived text
a1f98d3f9b16531214da740b47f90acb5a164f9a80d008cf343fbb4aeb0cf66d- Stable URL of this page
https://metraipolis.com/documento/dc_e35e92e50d99
How to cite
AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment — arXiv:2610.33644 — arXiv (en), https://arxiv.org/abs/2610.33644 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e35e92e50d99 (Metraipolis archive dc_e35e92e50d99, sha256 a1f98d3f9b16531214da740b47f90acb5a164f9a80d008cf343fbb4aeb0cf66d)
BibTeX
@misc{dce35e92e50d99,
title = {AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment — arXiv:2610.33644},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.33644},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e35e92e50d99; sha256 a1f98d3f9b16531214da740b47f90acb5a164f9a80d008cf343fbb4aeb0cf66d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33644. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Research ideas are always proposed under resource constraints: a plan must fit the available compute, API credit, and time. Yet evaluations of research-ideation agents mostly score unconstrained novelty or overlap with a published paper, and their LLM judges are rarely checked against peer review. We introduce AlphaIdeaBench, a benchmark of 100 ideation tasks (50 workshop calls for papers and 50 open problems stated in surveys), each paired with an explicit resource budget and a shared proposal schema. An LLM judge scores each proposal on 12 dimensions covering validity, novelty, and feasibility. To validate the judge, we compare its scores on four novelty axes (problem, method, scenario, and insight) with scores derived from reviewer comments on 600 papers from ICLR 2026, NeurIPS 2025, and ICML 2025. For problem and method novelty, Gemini 3.7 Flash is within one point of the reviewer-derived score on 94-97% of papers; agreement on scenario novelty is relatively lower. We evaluate eight published harnesses on a shared backbone and, within one harness, eight backbone LLMs. CoI-Agent obtains the highest overall score, narrowly ahead of SciMON, which leads on method and insight novelty. All harnesses score low on scenario novelty and rarely change the problem setting. The backbone largely determines which open problem a proposal targets, whereas the harness determines whether a proposal addresses one problem or combines several. Weaker backbones such as Llama 4 Scout fail to carry out that harness's strategy and score low on budget feasibility. Finally, when only the budget is varied on three topics, a larger budget changes the experimental setting on compute-intensive topics but only adds tests on an inexpensive one. We release the tasks, rubrics, and scores to foster standardized, reproducible benchmarking of future research-ideation agents under constraints. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.33644 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-27T07:55:30.653Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).