Metraipolis › archive › dc_ce01ac5bc615

Who is the Pronoun Pro? Comparing LLMs and Humans at Number-Disambiguation of 'they' — arXiv:2611.04968

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04968
License (as archived): CC0-1.0
Archived: 2026-11-04T21:40:16.548Z
SHA-256 of the archived text: a692ec563d96c45094bcf29231a5153ceaa7629b26c49aff562b23c243be3a5b
Stable URL of this page: https://metraipolis.com/documento/dc_ce01ac5bc615

How to cite

Who is the Pronoun Pro? Comparing LLMs and Humans at Number-Disambiguation of 'they' — arXiv:2611.04968 — arXiv (en), https://arxiv.org/abs/2611.04968 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ce01ac5bc615 (Metraipolis archive dc_ce01ac5bc615, sha256 a692ec563d96c45094bcf29231a5153ceaa7629b26c49aff562b23c243be3a5b)

BibTeX

@misc{dcce01ac5bc615,
  title = {Who is the Pronoun Pro? Comparing LLMs and Humans at Number-Disambiguation of 'they' — arXiv:2611.04968},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04968},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ce01ac5bc615; sha256 a692ec563d96c45094bcf29231a5153ceaa7629b26c49aff562b23c243be3a5b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04968. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This study evaluates whether large language models (LLMs) can reliably replace or complement human research assistants in linguistically complex annotation. In a controlled annotation task, 73 human annotators (local students and crowdworkers with and without a background in linguistics) and more than 40 LLM configurations number-disambiguated 174 authentic uses of 'they' from Reddit, including innovative and context-sensitive cases. The best LLM, Kimi K3, reached 94.5% accuracy, above the human majority vote (86.2%) and the mean of the top half of human annotators (78.5%); only the single best human annotator scored higher (95.7%). LLMs also offer substantial practical advantages in cost and speed. However, qualitative error analysis reveals a crucial complementarity: human reasoning in low-agreement cases exposes inconsistencies in the annotation typology and thereby improves the methodological framework itself. The study contributes (i) a benchmark-style human-LLM comparison for a semantically demanding task, (ii) transparent cost and time evidence, and (iii) a methodological argument against a purely replacement-oriented view of automation. LLMs are viable for large-scale annotation support in linguistics, but their responsible use requires human critical oversight. The dataset with its gold-standard labels, all human and LLM annotations, and the fully reproducible code are openly available as a Quarto-based, public repository.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2611.04968 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:35:49.698Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).