Metraipolis › archive › dc_8723143bb969

Can We Believe AI Peer Review? — arXiv:2610.19339

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19339
License (as archived)
CC0-1.0
Archived
2026-10-15T20:01:10.025Z
SHA-256 of the archived text
f4523453b1ef9bc9d0921249c86ce2f6b4922baf3c0b9c26c069dfe3bd7983af
Stable URL of this page
https://metraipolis.com/documento/dc_8723143bb969

How to cite

Can We Believe AI Peer Review? — arXiv:2610.19339 — arXiv (en), https://arxiv.org/abs/2610.19339 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8723143bb969 (Metraipolis archive dc_8723143bb969, sha256 f4523453b1ef9bc9d0921249c86ce2f6b4922baf3c0b9c26c069dfe3bd7983af)
BibTeX
@misc{dc8723143bb969,
  title = {Can We Believe AI Peer Review? — arXiv:2610.19339},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19339},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8723143bb969; sha256 f4523453b1ef9bc9d0921249c86ce2f6b4922baf3c0b9c26c069dfe3bd7983af},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19339. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.19339 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T19:35:48.214Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).