Metraipolis › archive › dc_9453842978bb

Data Annotation as Measurement — arXiv:2608.07297

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.07297
License (as archived)
CC0-1.0
Archived
2026-10-19T09:49:33.287Z
SHA-256 of the archived text
08830e5e06bc93e7988b3371949815cb6be9679cee4ae7841f3ce89d0c55455c
Stable URL of this page
https://metraipolis.com/documento/dc_9453842978bb

How to cite

Data Annotation as Measurement — arXiv:2608.07297 — arXiv (en), https://arxiv.org/abs/2608.07297 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9453842978bb (Metraipolis archive dc_9453842978bb, sha256 08830e5e06bc93e7988b3371949815cb6be9679cee4ae7841f3ce89d0c55455c)
BibTeX
@misc{dc9453842978bb,
  title = {Data Annotation as Measurement — arXiv:2608.07297},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.07297},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9453842978bb; sha256 08830e5e06bc93e7988b3371949815cb6be9679cee4ae7841f3ce89d0c55455c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07297. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern AI systems depend on annotated data, but annotation is rarely treated as the act of measurement that it is. Instead, annotation quality is commonly reduced to agreement: if multiple annotators assign the same annotation to a data instance, the annotations are taken to be high-quality. Yet agreement does not establish whether annotations validly capture the underlying concept they are meant to represent. In this paper, we argue that data annotation should be understood as a measurement problem. Like other forms of measurement, annotation requires defining a concept, operationalizing it through an instrument, applying that instrument, and evaluating the reliability and validity of the resulting measurements. Drawing on a literature review of annotation quality research (N=132) and semi-structured interviews with annotation team members (N=10), we develop a framework for diagnosing and correcting annotation issues. First, we map key decision points across annotation processes - including task design, annotator management, quality assessment, quality improvement, and adjudication - that shape annotation outcomes. Second, we identify five distinct sources of annotation issues: error, ambiguity, impossibility, subjectivity, and annotator identity. Annotation problems that appear similar at the level of outcomes often require different process-level interventions based on their sources. Finally, we translate measurement theory into practical guidance for annotation teams, showing how assessments of reliability and validity can move beyond agreement alone. By reframing annotation as measurement, we offer a conceptual foundation for improving the quality of annotated data used in AI research and practice.

Tema degli esploratori: governance digitale e beni comuni (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CY).
Categorie arXiv: Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2608.07297 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:48:35.569Z dalla pagina https://arxiv.org/catchup/cs.CY/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).