Metraipolis › archive › dc_0a90bbb7f765

Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus — arXiv:2609.13936

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.13936
License (as archived)
CC0-1.0
Archived
2026-10-15T23:14:51.363Z
SHA-256 of the archived text
66012a0b4bb62d02fd72e2929df52e74d25964276718ccb7bd9053231a54b4c6
Stable URL of this page
https://metraipolis.com/documento/dc_0a90bbb7f765

How to cite

Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus — arXiv:2609.13936 — arXiv (en), https://arxiv.org/abs/2609.13936 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0a90bbb7f765 (Metraipolis archive dc_0a90bbb7f765, sha256 66012a0b4bb62d02fd72e2929df52e74d25964276718ccb7bd9053231a54b4c6)
BibTeX
@misc{dc0a90bbb7f765,
  title = {Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus — arXiv:2609.13936},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.13936},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0a90bbb7f765; sha256 66012a0b4bb62d02fd72e2929df52e74d25964276718ccb7bd9053231a54b4c6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13936. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Datasets that ship automatically generated feature annotations invite a question rarely asked of them: would a human agree with those labels? This report answers that for the Objective Projection corpus, a Turkish narrative dataset whose scenes carry a per-scene applied_rules field from a rule-based detector over six craft features -- two prohibitions (emotion labelling, simile) and four positive techniques (materialized metaphor, micro-focus, temporal anchor, atmosphere contradiction). Three studies are reported. Study 1 ($n = 120$) scores the detector against blind labels from the scheme's own author. Study 2 ($n = 100$, a disjoint scene set) scores the detector plus Gemini 2.5 Flash and Grok against an independent non-expert rater whose labels were locked before any machine ran. Study 2b re-runs the identical protocol with Claude Fable 5 (High) and ChatGPT 5.5. The central result concerns one rule. On materialized metaphor -- closest to the methodology's theoretical core -- the five machine labellers returned positive rates of $0$, $1$, $40$, $72$ and $78$ out of $100$ scenes, against a human count of $9$. Cohen's $\kappa$ was at or indistinguishable from chance for five of six labellers, across both human references and both scene sets: $0.004$, $0.015$, $0.000$, $0.019$, $0.027$. Raw agreement ranged from $74.7\%$ to $84.5\%$, an artefact of class imbalance rather than a sign of competence. We deliberately do not resolve this into a single story. Two readings survive: the feature is genuinely inferential and beyond current automatic detection, or the rule's definition is not yet operational enough for any rater to apply consistently -- including the human. Distinguishing them needs a second independent human rater, which this report does not have and therefore does not claim.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.13936 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:51:09.851Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).