Metraipolis › archive › dc_a803cef2be1a

How Closely Do LLM Reviews Align with Human Peer Review? — arXiv:2608.03659

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03659
License (as archived)
CC0-1.0
Archived
2026-10-31T12:19:31.595Z
SHA-256 of the archived text
97bf8bf2fcb75c437013029e6291ac7eb214938bf603ad0a264b6504bd42951c
Stable URL of this page
https://metraipolis.com/documento/dc_a803cef2be1a

How to cite

How Closely Do LLM Reviews Align with Human Peer Review? — arXiv:2608.03659 — arXiv (en), https://arxiv.org/abs/2608.03659 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a803cef2be1a (Metraipolis archive dc_a803cef2be1a, sha256 97bf8bf2fcb75c437013029e6291ac7eb214938bf603ad0a264b6504bd42951c)
BibTeX
@misc{dca803cef2be1a,
  title = {How Closely Do LLM Reviews Align with Human Peer Review? — arXiv:2608.03659},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03659},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a803cef2be1a; sha256 97bf8bf2fcb75c437013029e6291ac7eb214938bf603ad0a264b6504bd42951c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03659. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting. We compare reviews from OpenAI GPT-5.4, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.6 with human reviews and final decisions for 300 topic-matched ICLR 2026 submissions, equally divided among oral, poster, and rejected papers. Each model reviewed every paper using identical instructions and rating scales after decision information was removed. Our study contributes a cross-provider analysis of three complementary dimensions: alignment with broad and fine-grained decision categories, differences in recommendation-scale usage, and thematic agreement in identified weaknesses. All three LLMs distinguished accepted from rejected papers, but none reproduced the oral versus poster distinction present in human ratings. Scoring patterns were provider-specific: Gemini assigned systematically higher ratings, while OpenAI and Claude were closer to humans for rejected and poster papers but more critical of oral papers. Human and LLM reviews also differed in emphasis, with LLMs more frequently identifying missing baseline comparisons and humans more often raising computational-efficiency concerns. These results show that broad decision alignment does not imply agreement with finer human judgments or reviewing priorities.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.03659 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:53.072Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).