Metraipolis › archive › dc_a88e16acaf4e
ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments — arXiv:2611.07489
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.07489
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:43:06.242Z
- SHA-256 of the archived text
c327168d6b21d9afe7316cf6aeef762e8aab014bdac6bca350821d29fabeec6a- Stable URL of this page
https://metraipolis.com/documento/dc_a88e16acaf4e
How to cite
ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments — arXiv:2611.07489 — arXiv (en), https://arxiv.org/abs/2611.07489 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a88e16acaf4e (Metraipolis archive dc_a88e16acaf4e, sha256 c327168d6b21d9afe7316cf6aeef762e8aab014bdac6bca350821d29fabeec6a)
BibTeX
@misc{dca88e16acaf4e,
title = {ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments — arXiv:2611.07489},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.07489},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a88e16acaf4e; sha256 c327168d6b21d9afe7316cf6aeef762e8aab014bdac6bca350821d29fabeec6a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07489. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judgments and their grounds. This defines a more honest mission for NLP in peer review than full automation. We present ReviewMap, a framework that classifies each comment as assessment of claims and scholarly standards or calibration of contribution value and venue fit, with finer-grained attributes applied conditionally by act type. Applied to 500 ICLR~2026 papers through multi-model LLM consensus, the act layer proves reliably automatable under the operational guideline (inter-model $\kappa \geq 0.75$), supported by five-human validation on 470 development comments, while the attribute layer requires human oversight, defining a measured automation boundary. At the corpus level, 17% of reviewer-paper instances contain no panel-coded calibration, and the panel-coded calibration share is nearly $3\times$ as high in Strengths as in Weaknesses. This boundary informs three affordances: a review guideline that operationalizes the assessment-then-calibration workflow for reviewers, a per-comment rebuttal strategy scaffold for authors, and a discussion-phase dashboard for area chairs. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Human-Computer Interaction (cs.HC). Pagina del lavoro: https://arxiv.org/abs/2611.07489 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-06T14:18:57.585Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).