Metraipolis › archive › dc_6a161d4974df
Same Sentence, Different Construct: Identity-Swapped Rationales Reveal Measurement Non-Invariance in LLM Hate Speech Classifiers — arXiv:2611.02491
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02491
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:23:41.037Z
- SHA-256 of the archived text
c657eabcf15d99f56dc9fa842b250b9311f0c809af39aa5c71d1e7b8b774bd52- Stable URL of this page
https://metraipolis.com/documento/dc_6a161d4974df
How to cite
Same Sentence, Different Construct: Identity-Swapped Rationales Reveal Measurement Non-Invariance in LLM Hate Speech Classifiers — arXiv:2611.02491 — arXiv (en), https://arxiv.org/abs/2611.02491 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6a161d4974df (Metraipolis archive dc_6a161d4974df, sha256 c657eabcf15d99f56dc9fa842b250b9311f0c809af39aa5c71d1e7b8b774bd52)
BibTeX
@misc{dc6a161d4974df,
title = {Same Sentence, Different Construct: Identity-Swapped Rationales Reveal Measurement Non-Invariance in LLM Hate Speech Classifiers — arXiv:2611.02491},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02491},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6a161d4974df; sha256 c657eabcf15d99f56dc9fa842b250b9311f0c809af39aa5c71d1e7b8b774bd52},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02491. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models are increasingly used as measurement instruments for socially consequential constructs such as hate speech, and are typically validated by their aggregate agreement with human labels. A valid instrument must also apply the same construct to every group it measures. We test this requirement using 421 \hc{} templates that are instantiated identically for seven protected groups, so that only the identity term varies. Four LLM classifiers each exceed 0.95 balanced accuracy, yet within identical templates two of them miss hateful statements about women at markedly higher rates than the same statements about other groups (excess false-negative rates of 8.3 and 4.4 percentage points), a gap first reported for fine-tuned classifiers in 2021. We show how the construct shifts. Comparing rationales within each template, we find that models invoke protected-group status for other groups but recast the identical statement about women as personal dislike or as aimed at an individual, in 22 of 30 missed templates for one model. Adding the benchmark's own definition, which lists sex as a protected characteristic, does not close the gap, and the contrast persists. Apparent false-positive disparities, by contrast, largely vanish under identity control and trace to reclaimed slurs present only for some groups. Errors shared by all models concentrate where model and benchmark definitions diverge, and pairwise counterfactual accuracy adds little beyond endpoint accuracy. We argue that validating LLM measurement requires identity-controlled tests of measurement invariance that examine both decisions and the criteria models state for them.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.02491 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:07.969Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).