Metraipolis › archive › dc_ae4564cc4b6f
Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories — arXiv:2611.02244
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02244
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:10:45.684Z
- SHA-256 of the archived text
aef315b9911b60e269c4851f4fb5a4c7f68201df1ad3ae2cbd74ba8c0a3c3614- Stable URL of this page
https://metraipolis.com/documento/dc_ae4564cc4b6f
How to cite
Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories — arXiv:2611.02244 — arXiv (en), https://arxiv.org/abs/2611.02244 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ae4564cc4b6f (Metraipolis archive dc_ae4564cc4b6f, sha256 aef315b9911b60e269c4851f4fb5a4c7f68201df1ad3ae2cbd74ba8c0a3c3614)
BibTeX
@misc{dcae4564cc4b6f,
title = {Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories — arXiv:2611.02244},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02244},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ae4564cc4b6f; sha256 aef315b9911b60e269c4851f4fb5a4c7f68201df1ad3ae2cbd74ba8c0a3c3614},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02244. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using PaperTrajectory-2026, the 2026 release of PaperTrajectory, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.02244 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:59:52.331Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).