Metraipolis › archive › dc_be6e829bb5fc
Do Language Models Know What Compelling Scientific Evidence Looks Like? — arXiv:2610.41015
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41015
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T07:42:35.053Z
- SHA-256 of the archived text
ecb890ca87a4f8d4aa86d553b5836a57728499e50239d7f0e37db89a4f55e6cc- Stable URL of this page
https://metraipolis.com/documento/dc_be6e829bb5fc
How to cite
Do Language Models Know What Compelling Scientific Evidence Looks Like? — arXiv:2610.41015 — arXiv (en), https://arxiv.org/abs/2610.41015 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_be6e829bb5fc (Metraipolis archive dc_be6e829bb5fc, sha256 ecb890ca87a4f8d4aa86d553b5836a57728499e50239d7f0e37db89a4f55e6cc)
BibTeX
@misc{dcbe6e829bb5fc,
title = {Do Language Models Know What Compelling Scientific Evidence Looks Like? — arXiv:2610.41015},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41015},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_be6e829bb5fc; sha256 ecb890ca87a4f8d4aa86d553b5836a57728499e50239d7f0e37db89a4f55e6cc},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41015. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Expert scientists excel at weighing empirical findings from multiple sources that vary in both statistical reliability and epistemic relevance. To what extent are language models, which are increasingly driving autonomous discovery systems, capable of such evidence deliberation? We introduce EDB-Bio, a scientific reasoning benchmark curated from real-world data in three biological domains: drug-perturbation transcriptomics, CRISPR screens, and human genome-wide association studies (GWAS). Each of EDB-Bio's 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. Throughout, quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. Our work extends calibration training and uncertainty analysis to a realistic scientific forecasting setting with applications to human cancer biology. We release all code, data, and trained models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.41015 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T07:39:37.355Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).