Metraipolis › archive › dc_f56540f9fcc2

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment — arXiv:2608.29696

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.29696
License (as archived)
CC0-1.0
Archived
2026-10-17T07:44:22.255Z
SHA-256 of the archived text
fb60d412b2ec08a5b3c839911da20b647db666637d0f0332fd4f6a443376e637
Stable URL of this page
https://metraipolis.com/documento/dc_f56540f9fcc2

How to cite

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment — arXiv:2608.29696 — arXiv (en), https://arxiv.org/abs/2608.29696 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f56540f9fcc2 (Metraipolis archive dc_f56540f9fcc2, sha256 fb60d412b2ec08a5b3c839911da20b647db666637d0f0332fd4f6a443376e637)
BibTeX
@misc{dcf56540f9fcc2,
  title = {Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment — arXiv:2608.29696},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.29696},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f56540f9fcc2; sha256 fb60d412b2ec08a5b3c839911da20b647db666637d0f0332fd4f6a443376e637},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29696. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. Ideation Arena evaluates ideas generated by 14 frontier LLMs and 5 research agent architectures built on 2 base models. To ensure a common starting point, Ideation Arena builds shared literature contexts from papers familiar to the participating researchers and provides the same contexts to all LLMs and agents. We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers and construct an Elo rating leaderboard of proposal-stage expert preferences in computer science under a shared closed-context protocol. We validate the rankings through interrater agreement and robustness analyses, showing that the leaderboard remains stable under changes in annotator composition and domain coverage. Our results show substantial variation in agent effectiveness, with some frameworks improving ideation quality over their backbones and others offering little benefit or even underperforming their base models. We further construct Ideation Arena Eval, a benchmark for assessing whether automated evaluators align with human preferences in research ideation. Experiments with current LLM judges show that they still cannot reliably reproduce expert preferences, with the best judge reaching 72.56% Soft Accuracy on Overall Quality. Our code, data, and leaderboards are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.29696 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.559Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).