Metraipolis › archive › dc_31647ddd2bf3
Social Bias or Social Desirability? Comparing Factorial Survey Experiments for Humans and Large Language Models — arXiv:2610.22409
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.22409
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:44:27.145Z
- SHA-256 of the archived text
c06ef0f1e4e8d47caad693bf9421cce49788570387323c96279386a7f820528c- Stable URL of this page
https://metraipolis.com/documento/dc_31647ddd2bf3
How to cite
Social Bias or Social Desirability? Comparing Factorial Survey Experiments for Humans and Large Language Models — arXiv:2610.22409 — arXiv (en), https://arxiv.org/abs/2610.22409 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_31647ddd2bf3 (Metraipolis archive dc_31647ddd2bf3, sha256 c06ef0f1e4e8d47caad693bf9421cce49788570387323c96279386a7f820528c)
BibTeX
@misc{dc31647ddd2bf3,
title = {Social Bias or Social Desirability? Comparing Factorial Survey Experiments for Humans and Large Language Models — arXiv:2610.22409},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.22409},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_31647ddd2bf3; sha256 c06ef0f1e4e8d47caad693bf9421cce49788570387323c96279386a7f820528c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22409. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Factorial survey experiments are a survey method from the social sciences that uses descriptions of socially complex situation (``vignettes'') to elicit judgments such as social bias. They provide two major benefits: (a), they reduce the effect of social desirability inherent in self-reported social biases, and (b), they make it possible to separate the effects of multiple sociodemographic attributes. These aspects are also desiderata for the evaluation of social bias in LLMs, but are often ignored in social bias evaluations. In our study, we evaluate social bias in humans and eleven LLMs on the basis of four different English and German factorial survey experiments. Our results show that the LLMs mirror the human biases only partially and sometimes even invert them. We provide evidence that this is due to a fundamental difference in how humans and LLMs react to socially complex situations. While such vignettes reduce social desirability bias in humans, this is not the case for LLMs, where the alignment-induced tendency towards socially desirable responses carries over directly. We therefore caution against applying bias measurement techniques from the social sciences that were developed for human to measure attitudes and bias in LLMs. Tema degli esploratori: governance digitale e beni comuni (elenco arXiv cs.CY, nuovi lavori del giorno). Categorie arXiv: Computers and Society (cs.CY). Pagina del lavoro: https://arxiv.org/abs/2610.22409 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:40:24.844Z dalla pagina https://arxiv.org/list/cs.CY/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).