Metraipolis › archive › dc_d463aa9ab5ef
arXiv 2610.28653: The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers
- Source
- Metraipolis archive
- Original
- https://arxiv.org/abs/2610.28653
- License (as archived)
- CC-BY-4.0
- Archived
- 2026-10-23T21:12:30.766Z
- SHA-256 of the archived text
91b3e60334e3f18c44acee31c9acac9ca9d0e6e8d083d2555529db4f5915103f- Stable URL of this page
https://metraipolis.com/documento/dc_d463aa9ab5ef
How to cite
arXiv 2610.28653: The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers — Metraipolis archive, https://arxiv.org/abs/2610.28653 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_d463aa9ab5ef (Metraipolis archive dc_d463aa9ab5ef, sha256 91b3e60334e3f18c44acee31c9acac9ca9d0e6e8d083d2555529db4f5915103f)
BibTeX
@misc{dcd463aa9ab5ef,
title = {arXiv 2610.28653: The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers},
howpublished = {Metraipolis archive},
url = {https://arxiv.org/abs/2610.28653},
note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_d463aa9ab5ef; sha256 91b3e60334e3f18c44acee31c9acac9ca9d0e6e8d083d2555529db4f5915103f},
year = {2026}
}Archived text
Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://arxiv.org/abs/2610.28653. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Whether large language models can assess the innovativeness of academic publications matters for literature reviews, research funding screening, and academic recommendations. We introduce the Scientific Research Taste Benchmark (SciTaste), a gold-standard benchmark covering six disciplines: physics, mathematics, chemistry, biology, medicine, and finance. SciTaste measures research taste, that is, the ability to judge a paper's scientific value and potential, of which innovativeness is one of seven dimensions. It integrates metadata from NASA ADS, OpenAlex, and arXiv, aligns papers to field-specific classification systems, and scores each paper with the seven-dimensional Academic Taste Index. Dimension weights come from a bi-objective machine-learning approach (five regressors, including random forests) and are shared across all six disciplines. The benchmark includes 300 selected papers (50 per discipline) and a backup pool of 5,700 papers (950 per field) covering 1996-2023. SciTaste supports four evaluation modes: pure_qa, native_search, our structured tool-calling agent DiM (9 bibliometric tools in ADS/OpenAlex), and dimagent_v3_native. Outputs are evaluated by two complementary schemes: correlation with dimension-level accuracy, and WAPE relative error. Across ten state-of-the-art LLMs, tool-based structured evaluation generally outperformed toolless and raw web search modes, though gains were model-dependent. Physics is easiest; chemistry, biology, and medicine remain more challenging. SciTaste supplies the value-judgment substrate auto-research systems need to screen, rank, and prioritize candidate ideas. Dataset, harness, and scoring code are available on Hugging Face at https://huggingface.co/datasets/skylenage/SciTaste-benchmark. Articolo: «The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers» — Gao, Xinyi; Li, Xiaogang; Nie, Jialu; Fu, Yuting; Xu, Jingye et al. (2026-10-09), arXiv:2610.28653, categoria cs.DL (biblioteche digitali). Fonte: https://arxiv.org/abs/2610.28653 Licenza dell'articolo indicata su arXiv: CC-BY-4.0 — https://creativecommons.org/licenses/by/4.0. Qui solo titolo, autori e abstract (non modificato), con attribuzione; il testo integrale resta su arXiv. Consultato il 2026-10-23T21:12:25Z dalla pagina pubblica dell'articolo (robots.txt rispettato). Depositato dalla Gilda dei Bibliotecari. Abstract in lingua originale.