Metraipolis › archive › dc_4f6cb005dca8
Expert-validated STEM QA — arXiv:2608.28591
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.28591
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:41:32.674Z
- SHA-256 of the archived text
b0eb7cdc30be7e513f27b30cb66e2d580e6779c390664510b1c168e566ef0e2e- Stable URL of this page
https://metraipolis.com/documento/dc_4f6cb005dca8
How to cite
Expert-validated STEM QA — arXiv:2608.28591 — arXiv (en), https://arxiv.org/abs/2608.28591 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4f6cb005dca8 (Metraipolis archive dc_4f6cb005dca8, sha256 b0eb7cdc30be7e513f27b30cb66e2d580e6779c390664510b1c168e566ef0e2e)
BibTeX
@misc{dc4f6cb005dca8,
title = {Expert-validated STEM QA — arXiv:2608.28591},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.28591},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4f6cb005dca8; sha256 b0eb7cdc30be7e513f27b30cb66e2d580e6779c390664510b1c168e566ef0e2e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28591. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recent advancements in AI are helping scientists achieve breakthroughs in fields such as mathematics, medicine, and materials sciences. New evaluation datasets for AI models contribute to such advancement in AI. In the STEM domain, frontier models have consumed most of the available online data, creating the need for human-created datasets that codify the knowledge of leading experts in the domain. There are several STEM datasets available for the research community in this field. However, there are some gaps in these datasets, leaving room for improvement. Examples of gaps include (1) saturation in model performance on these datasets, leaving no head-room for meaningful evaluations, (2) skewed taxonomy distributions, (3) multiple choice question format that is misaligned with how scientists use AI in the real world, and (4) inaccurate answers and rationales partially led by a contest-based data collection and a time-bound review process. In this study, we present 'Expert-validated STEM QA', a high-quality, expert-validated STEM dataset (N=398) in Physics, Chemistry, Biology, and Mathematics, created by 241 domain experts. We (1) carefully designed a taxonomy with balanced distribution, (2) vetted question contributors with quality-driven incentive, (3) conducted multiple rounds of reviews with revisions validated by domain experts based on consensus, and (4) created the dataset in verifiable question and answer format. Our study demonstrated low performance ($<25\%$) of frontier AI models on the dataset as a benchmark. Post-training on a separate, private version of the dataset (N=2,000) increased performance of the open source model by $15\%$ relative to the baseline model (p=0.045) on the STEM subset of HLE-verified dataset, indicating potential utility of the dataset for model training. We have open-sourced a portion of our dataset for the AI research community. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.28591 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.435Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).