Metraipolis › archive › dc_3536a35b293b

Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers (2026) · DOI 10.20944/preprints202609.1913.v1

Source
Metraipolis archive
Original
https://doi.org/10.20944/preprints202609.1913.v1
License (as archived)
CC-BY-4.0
Archived
2026-10-08T07:08:30.644Z
SHA-256 of the archived text
cabb895221a003afe133736ba458f4d9ea2008b06bcf2ffb849465e5114c029c
Stable URL of this page
https://metraipolis.com/documento/dc_3536a35b293b

How to cite

Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers (2026) · DOI 10.20944/preprints202609.1913.v1 — Metraipolis archive, https://doi.org/10.20944/preprints202609.1913.v1 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_3536a35b293b (Metraipolis archive dc_3536a35b293b, sha256 cabb895221a003afe133736ba458f4d9ea2008b06bcf2ffb849465e5114c029c)
BibTeX
@misc{dc3536a35b293b,
  title = {Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers (2026) · DOI 10.20944/preprints202609.1913.v1},
  howpublished = {Metraipolis archive},
  url = {https://doi.org/10.20944/preprints202609.1913.v1},
  note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_3536a35b293b; sha256 cabb895221a003afe133736ba458f4d9ea2008b06bcf2ffb849465e5114c029c},
  year = {2026}
}

Archived text

Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://doi.org/10.20944/preprints202609.1913.v1. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Articolo recente con licenza aperta: scheda dai metadati Crossref (posted-content).
Titolo: Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers
Autori: Sarah Elliott, Samantha Guitard, Angelika Ramos, Lisa Hartling
Pubblicato: 2026-09-22 · editore: MDPI AG
DOI: https://doi.org/10.20944/preprints202609.1913.v1
Licenza dell'articolo: CC-BY-4.0

Abstract (testo degli autori, CC-BY-4.0):
Background: There continues to be a growing number of systematic reviews with variable methodological quality. Using a systematic review to inform decision making relies on the quality of methods used to synthesize evidence. Tools that have been developed, such as AMSTAR 2, aim to assess the methodological quality and validity of systematic reviews of healthcare interventions. As this task takes time, use of large language models (LLMs) may help expedite the process. Objectives: We aim to test the ability of two LLMs, Claude Opus 5 and ChatGPT 5.5, to perform AMSTAR 2 assessments on a set of systematic reviews and to compare the accuracy to standard methods using human reviewers. As a secondary objective, we aim to explore the temporal consistency of the AMSTAR 2 assessments from each LLM over time (baseline and one month later).Methods: Our study sample includes open-access systematic reviews of interventions in pediatric populations. Our primary outcome is accuracy of each LLM’s AMSTAR 2 assessments as compared to a human reviewer process as the reference standard. Accuracy will be calculated as the total number of assessments of AMSTAR 2 items that correctly match the human reviewer team. Two humans will independently perform AMSTAR 2 assessments and come to consensus through discussion, or involvement with a third senior reviewer when required. A randomly selected subset of the systematic reviews will be used for the secondary objective exploring temporal consistency of the same LLM model versions over time. Results: AMSTAR 2 assessments will be performed on all 45 systematic reviews by each LLM and two human reviewers. Accuracy of the LLMs versus the human reference standard will be recorded for each AMSTAR 2 item. We will also identify whether there are common discrepancies for specific AMSTAR 2 items, and possible reasons (e.g., items requiring methodological judgement, vague methods reported in the systematic reviews, etc.). The temporal consistency for each model over time will be summarized narratively with descriptive statistics.Conclusion: Evaluating the methodological quality of systematic reviews with LLMs has the potential to more efficiently respond to the needs of decision-makers, by offsetting time spent performing AMSTAR 2 assessments, while maintaining accuracy. Our study will provide evidence to support or reject the use of LLMs to replace human reviewers in this process based on the accuracy of assessments, as well as whether responses are consistent over time. We will also identify AMSTAR 2 items that appear consistently problematic, where decision rules or adequate prompts are needed.

Fonte: https://api.crossref.org/works/10.20944%2Fpreprints202609.1913.v1 — metadati Crossref, consultati il 2026-10-08. Interrogazione: «evaluation of large language models».