Metraipolis › archive › dc_7e1744d330c8

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research — arXiv:2610.02202

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.02202
License (as archived): CC0-1.0
Archived: 2026-10-06T12:07:11.432Z
SHA-256 of the archived text: 3af57a48472297020957b4253553026c68896dede019e0ab5cab5d44738d67c2
Stable URL of this page: https://metraipolis.com/documento/dc_7e1744d330c8

How to cite

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research — arXiv:2610.02202 — arXiv (en), https://arxiv.org/abs/2610.02202 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7e1744d330c8 (Metraipolis archive dc_7e1744d330c8, sha256 3af57a48472297020957b4253553026c68896dede019e0ab5cab5d44738d67c2)

BibTeX

@misc{dc7e1744d330c8,
  title = {ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research — arXiv:2610.02202},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.02202},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7e1744d330c8; sha256 3af57a48472297020957b4253553026c68896dede019e0ab5cab5d44738d67c2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.02202. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2610.02202 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.004Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).