Metraipolis › archive › dc_fc7c92de095a
Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects — arXiv:2608.28626
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.28626
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:47:08.549Z
- SHA-256 of the archived text
4856e22dfb7d0b330ba2c9e6c4ada066738d7b5ff1429f4ec7ccaf72882f0fc3- Stable URL of this page
https://metraipolis.com/documento/dc_fc7c92de095a
How to cite
Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects — arXiv:2608.28626 — arXiv (en), https://arxiv.org/abs/2608.28626 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fc7c92de095a (Metraipolis archive dc_fc7c92de095a, sha256 4856e22dfb7d0b330ba2c9e6c4ada066738d7b5ff1429f4ec7ccaf72882f0fc3)
BibTeX
@misc{dcfc7c92de095a,
title = {Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects — arXiv:2608.28626},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.28626},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fc7c92de095a; sha256 4856e22dfb7d0b330ba2c9e6c4ada066738d7b5ff1429f4ec7ccaf72882f0fc3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28626. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly used to generate peer reviews, prompting examination of their capacity for critical evaluation. This study evaluates two multimodal LLMs, Qwen2.5-VL-72B and Pixtral-Large-124B, as reviewers across 165 submissions to the 2026 International Conference on Learning Representations, a venue that postdates both models' training cutoffs. Manuscripts were presented to both models with author identities blinded, replaced with high-prestige affiliations, or replaced with low-prestige affiliations, and in either text-only or text-with-figure format. Additionally, 145 verifiably detectable errors were inserted into 55 manuscripts to assess error identification under natural and verification-oriented prompts. Across all manuscript groups, including rejected submissions, LLM scores ranged from 7.0 to 8.1, whereas human mean scores ranged from 3.4 to 6.8. The models detected 12.1\% of the verified errors under natural prompting, and a one-sentence verification instruction increased detection to 22.2\%; however, 78\% of the errors remained undetected. Providing figures reduced error detection while increasing review scores. No visual error was reliably verified against its corresponding figure, and half of the text-only reviews described figures that were not provided. Author identity did not influence either review scores or error detection. LLM editorial decisions exactly matched those produced by simple score averaging. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.28626 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.665Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).