Metraipolis › archive › dc_e040ce1ba442
Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment — arXiv:2610.41391
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41391
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-02T07:42:27.302Z
- SHA-256 of the archived text
af60a05b8522ec7164e581280ababc03f41cf7367bda313a56b9ecb8346da9eb- Stable URL of this page
https://metraipolis.com/documento/dc_e040ce1ba442
How to cite
Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment — arXiv:2610.41391 — arXiv (en), https://arxiv.org/abs/2610.41391 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e040ce1ba442 (Metraipolis archive dc_e040ce1ba442, sha256 af60a05b8522ec7164e581280ababc03f41cf7367bda313a56b9ecb8346da9eb)
BibTeX
@misc{dce040ce1ba442,
title = {Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment — arXiv:2610.41391},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41391},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e040ce1ba442; sha256 af60a05b8522ec7164e581280ababc03f41cf7367bda313a56b9ecb8346da9eb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41391. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Growing submission volumes have renewed interest in using large language models (LLMs) to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it. A base-rate-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls. Abstract-only input beats full-paper summarization, and model choice matters as much as paradigm choice. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set's score spread, and finds none beating an always-reject baseline in accuracy. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Digital Libraries (cs.DL). Pagina del lavoro: https://arxiv.org/abs/2610.41391 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-02T07:37:15.437Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).