Metraipolis › archive › dc_5be0b58d01eb

Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026 — arXiv:2609.19420

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.19420
License (as archived)
CC0-1.0
Archived
2026-10-10T04:00:26.908Z
SHA-256 of the archived text
9891dc1c2b1e070b3afd5e8ca0b96dcd0abf24f5c12bbb3a3c6e99b250165392
Stable URL of this page
https://metraipolis.com/documento/dc_5be0b58d01eb

How to cite

Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026 — arXiv:2609.19420 — arXiv (en), https://arxiv.org/abs/2609.19420 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5be0b58d01eb (Metraipolis archive dc_5be0b58d01eb, sha256 9891dc1c2b1e070b3afd5e8ca0b96dcd0abf24f5c12bbb3a3c6e99b250165392)
BibTeX
@misc{dc5be0b58d01eb,
  title = {Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026 — arXiv:2609.19420},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.19420},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5be0b58d01eb; sha256 9891dc1c2b1e070b3afd5e8ca0b96dcd0abf24f5c12bbb3a3c6e99b250165392},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19420. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLMs are rapidly reshaping peer review, making it important to understand how reviewers use them in practice and how different LLM-use policies affect review outcomes. We investigate these questions through a randomized experiment and an anonymous post-survey at ICML 2026, a major machine learning conference involving over 24,000 papers and 17,000 reviewers. Reviewers were assigned to either a conservative policy prohibiting all LLM use or a permissive policy allowing limited assistance, with randomization among a subset of main-track papers and reviewers. Policy assignment had near-zero effects on final paper decisions, paper scores, and reviewer confidence, although reviews under the permissive policy were 5.5-7% longer. Post-survey responses (N=1,486) revealed diverse attitudes toward LLMs and substantial noncompliance: 22.5% of conservative-policy reviewers reported using an LLM despite the prohibition, and 36.5% of permissive-policy reviewers reported at least one explicitly disallowed use. We discuss implications for future peer-review policy and tool design.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.HC).
Categorie arXiv: Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2609.19420 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:28:23.621Z dalla pagina https://arxiv.org/catchup/cs.HC/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).