Metraipolis › archive › dc_5454166f970e
Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation — arXiv:2609.35824
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.35824
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:56:20.081Z
- SHA-256 of the archived text
9e175137e9f6f5a44c2f5f82863b16b97066b3e3e8f23d3d5e8f8ab7b7615ef8- Stable URL of this page
https://metraipolis.com/documento/dc_5454166f970e
How to cite
Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation — arXiv:2609.35824 — arXiv (en), https://arxiv.org/abs/2609.35824 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5454166f970e (Metraipolis archive dc_5454166f970e, sha256 9e175137e9f6f5a44c2f5f82863b16b97066b3e3e8f23d3d5e8f8ab7b7615ef8)
BibTeX
@misc{dc5454166f970e,
title = {Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation — arXiv:2609.35824},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.35824},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5454166f970e; sha256 9e175137e9f6f5a44c2f5f82863b16b97066b3e3e8f23d3d5e8f8ab7b7615ef8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35824. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' $\kappa = 0.91$). Agreement fell when we changed the task design for the same tweets (median Cohen's $\kappa = 0.76$). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Methodology (stat.ME). Pagina del lavoro: https://arxiv.org/abs/2609.35824 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:27:03.160Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).