Metraipolis › archive › dc_fa00011efec3
Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals — arXiv:2610.31414
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.31414
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-23T15:16:40.556Z
- SHA-256 of the archived text
9b949daf004f37f72c8b6656b8e6c4c1fb94cf222a3afd6021c32f80091ed561- Stable URL of this page
https://metraipolis.com/documento/dc_fa00011efec3
How to cite
Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals — arXiv:2610.31414 — arXiv (en), https://arxiv.org/abs/2610.31414 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa00011efec3 (Metraipolis archive dc_fa00011efec3, sha256 9b949daf004f37f72c8b6656b8e6c4c1fb94cf222a3afd6021c32f80091ed561)
BibTeX
@misc{dcfa00011efec3,
title = {Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals — arXiv:2610.31414},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.31414},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa00011efec3; sha256 9b949daf004f37f72c8b6656b8e6c4c1fb94cf222a3afd6021c32f80091ed561},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31414. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
With the growth of scientific output, the peer review system is approaching a breaking point: reviewers are increasingly hard to secure, and turnaround times lengthen accordingly. LLM-assisted or LLM-generated peer review has been proposed as a remedy, but how such models would treat manuscripts already published in top-tier journals, the closest thing the current system has to a gold standard of scientific validity, is not known. Here we assembled a corpus of 200 articles, each released under a CC BY 4.0 license and published in 2026, from 20 general-science, life-science, and clinical-medicine journals including Nature, Science, and Cell, and asked four current LLMs (two proprietary, two open-weight) to produce independent peer-review reports of each one under a blinded protocol. Their decisions were overwhelmingly harsh and starkly inconsistent with each other: 605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100% (p < 0.001), disagreement on manuscripts that had, by construction, already survived real editorial review at the journals in question. When the same manuscripts were doctored with one deliberately planted, field-specific methodological or statistical flaw, detection rates varied from 17.0% to 60.5% across models (p < 0.001). Recommending revision for papers that had already passed editorial review (low specificity) while missing flaws deliberately placed in front of them (low sensitivity), these models produce output that should be read for what it is: an uneven, framing- and model-dependent signal. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Digital Libraries (cs.DL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.31414 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-23T15:09:08.621Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).