Metraipolis › archive › dc_088ca60cdc85

Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts — arXiv:2610.10318

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.10318
License (as archived): CC0-1.0
Archived: 2026-10-08T11:26:21.852Z
SHA-256 of the archived text: 76e744629866b2d2138ea207d78857e91521e34e410c01fddec5745f2801a529
Stable URL of this page: https://metraipolis.com/documento/dc_088ca60cdc85

How to cite

Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts — arXiv:2610.10318 — arXiv (en), https://arxiv.org/abs/2610.10318 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_088ca60cdc85 (Metraipolis archive dc_088ca60cdc85, sha256 76e744629866b2d2138ea207d78857e91521e34e410c01fddec5745f2801a529)

BibTeX

@misc{dc088ca60cdc85,
  title = {Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts — arXiv:2610.10318},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.10318},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_088ca60cdc85; sha256 76e744629866b2d2138ea207d78857e91521e34e410c01fddec5745f2801a529},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10318. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Social media is a rich source of real-time public sentiment, but widely used sentiment analysis tools are often applied without understanding their limitations. In this study, we evaluate the inter-rater reliability of three bespoke sentiment analysis tools (TextBlob, VADER, and Twitter-roBERTa-base) and three large language models (LLMs: Qwen3-32B, GPT-OSS-120B, Llama-4-Maverick-17B) against six human raters across 100 tweets. We measured agreement using two statistical measures: Cohen's kappa for pairwise comparisons and Fleiss' kappa for multiple raters. Even among the human raters, our results showed only fair agreement, highlighting the subjectivity of sentiment analysis. Higher agreement was observed under the binary sentiment classification (negative vs. non-negative and positive vs. non-positive) than under the three-class classification across both humans and automated tools. The Twitter-roBERTa-base model showed the strongest alignment with human ratings, outperforming both bespoke sentiment tools and LLMs, particularly in distinguishing negative versus non-negative sentiment. LLMs showed substantial agreement among themselves and moderate to substantial alignment with humans, performing better in positive vs. non-positive classifications. Our findings underscore that domain-specific fine-tuning remains crucial for reliable social media sentiment analysis, and human-centered evaluation remains essential for establishing gold-standard labels.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.10318 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:24:06.350Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).