Metraipolis › archive › dc_d693bf4e9e22

Do Language Models Know What Compelling Scientific Evidence Looks Like? — arXiv:2610.41015

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41015
License (as archived): CC0-1.0
Archived: 2026-11-03T05:32:36.984Z
SHA-256 of the archived text: b3cec3f3a28af858ce344139aa38b6aa4a4d064767a6abbd3e129e475c11dd1c
Stable URL of this page: https://metraipolis.com/documento/dc_d693bf4e9e22

How to cite

Do Language Models Know What Compelling Scientific Evidence Looks Like? — arXiv:2610.41015 — arXiv (en), https://arxiv.org/abs/2610.41015 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d693bf4e9e22 (Metraipolis archive dc_d693bf4e9e22, sha256 b3cec3f3a28af858ce344139aa38b6aa4a4d064767a6abbd3e129e475c11dd1c)

BibTeX

@misc{dcd693bf4e9e22,
  title = {Do Language Models Know What Compelling Scientific Evidence Looks Like? — arXiv:2610.41015},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41015},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d693bf4e9e22; sha256 b3cec3f3a28af858ce344139aa38b6aa4a4d064767a6abbd3e129e475c11dd1c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41015. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Expert scientists excel at weighing empirical findings from multiple sources that vary in both statistical reliability and epistemic relevance. To what extent are language models, which are increasingly driving autonomous discovery systems, capable of such evidence deliberation? We introduce EDB-Bio, a scientific reasoning benchmark curated from real-world data in three biological domains: drug-perturbation transcriptomics, CRISPR screens, and human genome-wide association studies (GWAS). Each of EDB-Bio's 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. Throughout, quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. Our work extends calibration training and uncertainty analysis to a realistic scientific forecasting setting with applications to human cancer biology. We release all code, data, and trained models.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.41015 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T05:18:52.187Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).