Metraipolis › archive › dc_ddabae65651f

Evaluating LLMs as Judges of Counterspeech Quality Against Online Hate Speech — arXiv:2611.04627

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04627
License (as archived)
CC0-1.0
Archived
2026-11-04T18:38:43.311Z
SHA-256 of the archived text
6a54105be722ecf12ebd01d314086f0b5d54c6f240e5bc228234b3bad9b786d5
Stable URL of this page
https://metraipolis.com/documento/dc_ddabae65651f

How to cite

Evaluating LLMs as Judges of Counterspeech Quality Against Online Hate Speech — arXiv:2611.04627 — arXiv (en), https://arxiv.org/abs/2611.04627 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ddabae65651f (Metraipolis archive dc_ddabae65651f, sha256 6a54105be722ecf12ebd01d314086f0b5d54c6f240e5bc228234b3bad9b786d5)
BibTeX
@misc{dcddabae65651f,
  title = {Evaluating LLMs as Judges of Counterspeech Quality Against Online Hate Speech — arXiv:2611.04627},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04627},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ddabae65651f; sha256 6a54105be722ecf12ebd01d314086f0b5d54c6f240e5bc228234b3bad9b786d5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04627. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Prolonged exposure to hate speech on social media can cause lasting psychological distress among users. Counterspeech has emerged as an effective, non-censorial mitigation strategy, but its deployment requires rigorous quality assessment. Manual assessment is unscalable due to the large volume of content on the Web and imposes severe emotional strain on human annotators. Large Language Models (LLMs) offer a scalable alternative, yet concerns remain regarding their reliability. In this paper, we investigate the extent to which LLMs can effectively perform this task. We test 12 open-weight LLMs of different sizes and families under zero-shot, few-shot, and chain-of-thought prompting, to approximate pointwise and pairwise human expert assessment of counterspeech quality across four dimensions: relevance, coherence, aggressiveness, and suitableness. We find that (i) LLMs show low agreement and moderate rank correlations with human experts: our best configuration, Qwen 2.5 72B on coherence, achieves up to a linear-weighted Cohen's $\kappa_l = 0.19$ and Spearman's $\rho = 0.42$ in pointwise evaluation and an unweighted Cohen's $\kappa = 0.25$ in pairwise evaluation. (ii) Converting pointwise scores into pairwise labels substantially improves human--LLM agreement for selected configurations. For same-instance coherence, Qwen 2.5 72B under zero-shot prompting achieves $\kappa = 0.35$ with converted labels, compared with $\kappa = 0.25$ when judging pairs directly, and Mixtral 8$\times$7B under few-shot prompting improves from $\kappa = 0.07$ to $\kappa = 0.34$. (iii) LLMs agree more with one another than with human experts, particularly in pairwise evaluation, where the strongest model pair, OLMo 2 32B and Llama 3.3 70B on coherence, reaches substantial agreement with $\kappa = 0.76$, whereas their agreement with the experts in the same configuration does not exceed $\kappa = 0.21$.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04627 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:52.155Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).