Metraipolis › archive › dc_e040ce1ba442

Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment — arXiv:2610.41391

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41391
License (as archived)
CC0-1.0
Archived
2026-11-02T07:42:27.302Z
SHA-256 of the archived text
af60a05b8522ec7164e581280ababc03f41cf7367bda313a56b9ecb8346da9eb
Stable URL of this page
https://metraipolis.com/documento/dc_e040ce1ba442

How to cite

Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment — arXiv:2610.41391 — arXiv (en), https://arxiv.org/abs/2610.41391 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e040ce1ba442 (Metraipolis archive dc_e040ce1ba442, sha256 af60a05b8522ec7164e581280ababc03f41cf7367bda313a56b9ecb8346da9eb)
BibTeX
@misc{dce040ce1ba442,
  title = {Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment — arXiv:2610.41391},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41391},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e040ce1ba442; sha256 af60a05b8522ec7164e581280ababc03f41cf7367bda313a56b9ecb8346da9eb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41391. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Growing submission volumes have renewed interest in using large language models (LLMs) to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it. A base-rate-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls. Abstract-only input beats full-paper summarization, and model choice matters as much as paradigm choice. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set's score spread, and finds none beating an always-reject baseline in accuracy. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Digital Libraries (cs.DL).
Pagina del lavoro: https://arxiv.org/abs/2610.41391 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:37:15.437Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).