Metraipolis › archive › dc_98c2ef7bb5a7

When F1 Is Not Enough: Auditing Codebooks for LLM-Assisted Annotation — arXiv:2611.08846

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.08846
License (as archived)
CC0-1.0
Archived
2026-11-09T22:33:46.403Z
SHA-256 of the archived text
07969e880702d1d3a3e85379ed44a3177b1a1cbfc8a315a8811ace47f5dd6ad6
Stable URL of this page
https://metraipolis.com/documento/dc_98c2ef7bb5a7

How to cite

When F1 Is Not Enough: Auditing Codebooks for LLM-Assisted Annotation — arXiv:2611.08846 — arXiv (en), https://arxiv.org/abs/2611.08846 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_98c2ef7bb5a7 (Metraipolis archive dc_98c2ef7bb5a7, sha256 07969e880702d1d3a3e85379ed44a3177b1a1cbfc8a315a8811ace47f5dd6ad6)
BibTeX
@misc{dc98c2ef7bb5a7,
  title = {When F1 Is Not Enough: Auditing Codebooks for LLM-Assisted Annotation — arXiv:2611.08846},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08846},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_98c2ef7bb5a7; sha256 07969e880702d1d3a3e85379ed44a3177b1a1cbfc8a315a8811ace47f5dd6ad6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08846. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) are increasingly used for text annotation in social science, but standard performance metrics do not explain why errors occur. Low F1 may reflect limitations of the annotation system, defects in the written codebook, or inconsistency in reference labels. We propose a pre-deployment codebook audit for LLM-assisted annotation. The audit uses class-specific metrics and confusion matrices to identify problematic classes and boundaries, structured document-level review to attribute error sources, and targeted codebook revision followed by held-out evaluation. We apply the audit to two published multi-class codebooks from distinct domains. In both applications, the audit identifies under-specified or conflicting coding rules, and targeted revisions reduce the diagnosed errors on held-out data. Analyses with two LLM annotators show that some problematic boundaries and codebook revisions transfer across models, whereas other revision effects are annotator-specific. The results show that LLM annotation errors can help identify weaknesses in the written measurement instrument, not only in the model. Codebook auditing complements prompting, retrieval, and fine-tuning by clarifying whether persistent errors call for model improvement, codebook revision, or renewed scrutiny of reference labels.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.08846 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.257Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).