Metraipolis › archive › dc_08da7987a5ea

The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025) — arXiv:2610.06591

Source: arXiv
Original: https://arxiv.org/abs/2610.06591
License (as archived): CC0-1.0
Archived: 2026-10-06T05:25:40.762Z
SHA-256 of the archived text: 8ea65f74b5435c4d424cb4cb99ffd8f4577178d9f7c625468a41a2a2d7cadf85
Stable URL of this page: https://metraipolis.com/documento/dc_08da7987a5ea

How to cite

The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025) — arXiv:2610.06591 — arXiv, https://arxiv.org/abs/2610.06591 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_08da7987a5ea (Metraipolis archive dc_08da7987a5ea, sha256 8ea65f74b5435c4d424cb4cb99ffd8f4577178d9f7c625468a41a2a2d7cadf85)

BibTeX

@misc{dc08da7987a5ea,
  title = {The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025) — arXiv:2610.06591},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06591},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_08da7987a5ea; sha256 8ea65f74b5435c4d424cb4cb99ffd8f4577178d9f7c625468a41a2a2d7cadf85},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06591. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.06591 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.699Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).