Metraipolis › archive › dc_fc897763f8ee

IdeaTaste: Do LLMs Share the Research Taste of Human Researchers? — arXiv:2611.03424

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03424
License (as archived)
CC0-1.0
Archived
2026-11-03T18:23:52.481Z
SHA-256 of the archived text
afd185200e05be5f9aa214b9fc49392c499b4df62a30031087f71ce6be414dd9
Stable URL of this page
https://metraipolis.com/documento/dc_fc897763f8ee

How to cite

IdeaTaste: Do LLMs Share the Research Taste of Human Researchers? — arXiv:2611.03424 — arXiv (en), https://arxiv.org/abs/2611.03424 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fc897763f8ee (Metraipolis archive dc_fc897763f8ee, sha256 afd185200e05be5f9aa214b9fc49392c499b4df62a30031087f71ce6be414dd9)
BibTeX
@misc{dcfc897763f8ee,
  title = {IdeaTaste: Do LLMs Share the Research Taste of Human Researchers? — arXiv:2611.03424},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03424},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fc897763f8ee; sha256 afd185200e05be5f9aa214b9fc49392c499b4df62a30031087f71ce6be414dd9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03424. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard ( this https URL ) that benchmarks contemporary AI proposals against future publication cycles.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.03424 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:07.996Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).