Metraipolis › archive › dc_ecd23ae5bae7

The Structure of Disagreement in Emotion Annotation — arXiv:2611.01539

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01539
License (as archived)
CC0-1.0
Archived
2026-11-03T18:22:53.113Z
SHA-256 of the archived text
59cfaf614111008ff98379268e9a5ea4b3de426167a92cec5e8079d8d819d107
Stable URL of this page
https://metraipolis.com/documento/dc_ecd23ae5bae7

How to cite

The Structure of Disagreement in Emotion Annotation — arXiv:2611.01539 — arXiv (en), https://arxiv.org/abs/2611.01539 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ecd23ae5bae7 (Metraipolis archive dc_ecd23ae5bae7, sha256 59cfaf614111008ff98379268e9a5ea4b3de426167a92cec5e8079d8d819d107)
BibTeX
@misc{dcecd23ae5bae7,
  title = {The Structure of Disagreement in Emotion Annotation — arXiv:2611.01539},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01539},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ecd23ae5bae7; sha256 59cfaf614111008ff98379268e9a5ea4b3de426167a92cec5e8079d8d819d107},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01539. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Calculating inter-annotator agreement is an established practice in NLP, and disagreement-aware work increasingly retains the distribution of human judgments rather than reducing annotations to a single gold label. Keeping the distribution, however, does not explain why annotators disagree. This paper examines the types of disagreement that occur in emotion annotation, starting from multilabel annotations of 1,677 sentence-level units from Ernest Hemingway's The Old Man and the Sea and testing the patterns they reveal against individual-level annotations from BRIGHTER and GoEmotions. Across the three datasets, we find three main recurring disagreement structures. Emotion categories overlap systematically; annotators differ both in how broadly they apply labels and in how they treat particular category boundaries, while sparse participation and explicit abstention can look like disagreement. Where intensity is rated, disagreement over whether an emotion is present can be separated from disagreement over its intensity once recognized. Similar aggregate disagreement can therefore arise from qualitatively different annotation processes. Hard aggregation removes most of this structure, while annotator vote-share representations keep marginal label frequencies but not the co-occurrence, annotator-specific, or procedural patterns that produced them. We argue that disagreement should be understood in terms of its underlying patterns before it is summarized, aggregated, or used as a modeling target, and propose a vocabulary for disagreement structure.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.01539 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:07.932Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).