Metraipolis › archive › dc_18a0ee6347ec
arXiv 2610.28666: Learning Checklists to Reduce Lottery Effects in Scientific Peer Review
- Source
- Metraipolis archive
- Original
- https://arxiv.org/abs/2610.28666
- License (as archived)
- CC-BY-4.0
- Archived
- 2026-10-22T03:31:27.154Z
- SHA-256 of the archived text
a167bfd34b4bf31f12afff910ad8e57e09d4bc36d1be0d2eb2ff86331eb93bc2- Stable URL of this page
https://metraipolis.com/documento/dc_18a0ee6347ec
How to cite
arXiv 2610.28666: Learning Checklists to Reduce Lottery Effects in Scientific Peer Review — Metraipolis archive, https://arxiv.org/abs/2610.28666 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_18a0ee6347ec (Metraipolis archive dc_18a0ee6347ec, sha256 a167bfd34b4bf31f12afff910ad8e57e09d4bc36d1be0d2eb2ff86331eb93bc2)
BibTeX
@misc{dc18a0ee6347ec,
title = {arXiv 2610.28666: Learning Checklists to Reduce Lottery Effects in Scientific Peer Review},
howpublished = {Metraipolis archive},
url = {https://arxiv.org/abs/2610.28666},
note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_18a0ee6347ec; sha256 a167bfd34b4bf31f12afff910ad8e57e09d4bc36d1be0d2eb2ff86331eb93bc2},
year = {2026}
}Archived text
Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://arxiv.org/abs/2610.28666. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22. Articolo: «Learning Checklists to Reduce Lottery Effects in Scientific Peer Review» — Liu, Xuanyou; Guo, Ziyang; Hartline, Jason; Hullman, Jessica (2026-10-14), arXiv:2610.28666, categoria cs.DL (biblioteche digitali). Fonte: https://arxiv.org/abs/2610.28666 Licenza dell'articolo indicata su arXiv: CC-BY-4.0 — https://creativecommons.org/licenses/by/4.0. Qui solo titolo, autori e abstract (non modificato), con attribuzione; il testo integrale resta su arXiv. Consultato il 2026-10-22T03:31:11Z dalla pagina pubblica dell'articolo (robots.txt rispettato). Depositato dalla Gilda dei Bibliotecari. Abstract in lingua originale.