Metraipolis › archive › dc_484beeba839a

Kurate: Scalable Scientific Quality Analysis — arXiv:2610.07306

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.07306
License (as archived)
CC0-1.0
Archived
2026-10-07T11:23:02.249Z
SHA-256 of the archived text
08a1e276cbe5b7e265ac645d6a1a2849e7084ab39575d79dd7d8592a317bb6ee
Stable URL of this page
https://metraipolis.com/documento/dc_484beeba839a

How to cite

Kurate: Scalable Scientific Quality Analysis — arXiv:2610.07306 — arXiv (en), https://arxiv.org/abs/2610.07306 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_484beeba839a (Metraipolis archive dc_484beeba839a, sha256 08a1e276cbe5b7e265ac645d6a1a2849e7084ab39575d79dd7d8592a317bb6ee)
BibTeX
@misc{dc484beeba839a,
  title = {Kurate: Scalable Scientific Quality Analysis — arXiv:2610.07306},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07306},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_484beeba839a; sha256 08a1e276cbe5b7e265ac645d6a1a2849e7084ab39575d79dd7d8592a317bb6ee},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07306. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Scientific search systems can find papers that are relevant to a question, but they generally do not assess the quality of the evidence that those papers provide. We present Kurate, a system that uses large language models (LLMs) to assess the quality of published studies. Kurate uses both the paper and its related documents (e.g., the study's trial registration and protocol), and links each of its judgments to the passage of text on which that judgment is based. We applied Kurate to a corpus of 4,347 papers (3,913 of which report randomized trials) and scored each paper on 8 dimensions of study design and reporting: specifically, statistical power, causal identification, preregistration, selective reporting, measurement validity, analysis prespecification, reporting transparency, and conflict of interest and funding. Across the corpus, we found that papers most often exhibited issues with statistical power, selective reporting, and analysis prespecification, although average quality differed between clinical areas. When compared against expert annotations of 60 held-out clinical-trial documents, the information Kurate extracted matched the expert label in 221/242 protocol scorepoints and 294/370 results-publication scorepoints, with AC1 0.94 and 0.81, respectively. Using a well-reputed, high quality clinical trial as a worked example, we show how a single paper's overall grade breaks down into separate judgments, with each linked to specific evidence from the trial's registration, protocol, and published report. Together, these results show that large-scale quality assessment of this kind is feasible, and that it can be used to address meta-scientific research questions.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.07306 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:39.414Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).