Metraipolis › archive › dc_b97809a32dc5

Does ChatGPT score research quality differently by gender? — arXiv:2608.09552

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.09552
License (as archived)
CC0-1.0
Archived
2026-10-19T08:57:03.685Z
SHA-256 of the archived text
30c9aad3573374d524e2e184fd8cff53652f8c70c2dce2e9403e61c641072f3d
Stable URL of this page
https://metraipolis.com/documento/dc_b97809a32dc5

How to cite

Does ChatGPT score research quality differently by gender? — arXiv:2608.09552 — arXiv (en), https://arxiv.org/abs/2608.09552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b97809a32dc5 (Metraipolis archive dc_b97809a32dc5, sha256 30c9aad3573374d524e2e184fd8cff53652f8c70c2dce2e9403e61c641072f3d)
BibTeX
@misc{dcb97809a32dc5,
  title = {Does ChatGPT score research quality differently by gender? — arXiv:2608.09552},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.09552},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b97809a32dc5; sha256 30c9aad3573374d524e2e184fd8cff53652f8c70c2dce2e9403e61c641072f3d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models (LLMs) are being considered for research evaluation, raising concerns about the introduction of AI bias. This study investigates whether ChatGPT research quality scores differ by first-author gender using 89,744 journal articles from the UK Research Excellence Framework (REF) 2021. Author information was withheld from ChatGPT to avoid direct gender bias. Nevertheless, male first-authored papers had slightly higher ChatGPT scores in most Units of Assessment (UoAs), especially in health, science and engineering-related subjects, and this pattern was often stronger for ChatGPT than for REF scores, based on a departmental-level proxy. Rank-based ChatGPT gains relative to REF scores were also more favourable for male first-authored papers in most UoAs, although the differences were generally small. Gender differences were not evident for solo research in the social sciences, arts and humanities, however. The male-favouring pattern for first-authored research was not explained by gender differences in writing styles, at least as reflected in abstract complexity. Some ChatGPT-REF differences may also reflect the departmental averaging process used to generate the REF proxy scores. Average ChatGPT scores may differ by first-author gender indirectly through other factors, such as field, topic, method, journal context or authorship structure. Thus, this is an additional reason to be cautious with AI-based research evaluation.

Tema degli esploratori: ricerca e scienza aperta (retro-catalogo arXiv: annunci del 2026-08-11 in cs.DL).
Categorie arXiv: Digital Libraries (cs.DL).
Pagina del lavoro: https://arxiv.org/abs/2608.09552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:53:22.439Z dalla pagina https://arxiv.org/catchup/cs.DL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).