Metraipolis › archive › dc_2280ee43e0ae

Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics — arXiv:2609.23264

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.23264
License (as archived)
CC0-1.0
Archived
2026-10-09T06:49:47.890Z
SHA-256 of the archived text
2ba5ed8972bd6eb7f9fdeb29782b1d482b8777ee00e50b1c2bd94ba29c729eb9
Stable URL of this page
https://metraipolis.com/documento/dc_2280ee43e0ae

How to cite

Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics — arXiv:2609.23264 — arXiv (en), https://arxiv.org/abs/2609.23264 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2280ee43e0ae (Metraipolis archive dc_2280ee43e0ae, sha256 2ba5ed8972bd6eb7f9fdeb29782b1d482b8777ee00e50b1c2bd94ba29c729eb9)
BibTeX
@misc{dc2280ee43e0ae,
  title = {Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics — arXiv:2609.23264},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.23264},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2280ee43e0ae; sha256 2ba5ed8972bd6eb7f9fdeb29782b1d482b8777ee00e50b1c2bd94ba29c729eb9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23264. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high score because it is fluent, organized, and polished, rather than because it provides a strong evaluation of the paper. This risk is especially important in AI-assisted reviewing, where reviewers may use LLMs to improve clarity or presentation while preserving the underlying judgments. We propose a statistical framework for testing whether peer-review evaluation metrics capture substantive review quality beyond surface-level linguistic form. The framework compares original human reviews with faithful LLM rewrites that preserve the same evaluative content while changing wording and presentation. Using a dataset comprising 4,044 meaning-preserving rewrites derived from 674 human reviews from ICLR and NeurIPS, we evaluate 29 content-oriented peer-review evaluation metrics drawn from four prior works through complementary tests of surface sensitivity and robustness. Although these metrics are intended to capture review properties beyond surface-level, writing-dependent characteristics, we find that sensitivity to rewriting is widespread. Under our primary analysis, 23 metrics assign significantly different scores to reviews whose evaluative content is preserved, while only six satisfy our robustness criterion. The patterns are largely consistent across two LLM judge models, suggesting that the issue is not specific to a single judge. These findings show that many peer-review evaluation metrics partially conflate review quality with linguistic presentation, and indicate that robustness to meaning-preserving rewriting should be validated before such metrics are used to compare human-written, AI-assisted, and AI-generated reviews.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.23264 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:44.985Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).