Metraipolis › archive › dc_f2722107fca3
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents — arXiv:2608.14606
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14606
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T17:47:49.312Z
- SHA-256 of the archived text
1c8d7cdd010849b3af9cedf8f3594bee9d039bc315c5d96758b8729123c61333- Stable URL of this page
https://metraipolis.com/documento/dc_f2722107fca3
How to cite
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents — arXiv:2608.14606 — arXiv (en), https://arxiv.org/abs/2608.14606 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f2722107fca3 (Metraipolis archive dc_f2722107fca3, sha256 1c8d7cdd010849b3af9cedf8f3594bee9d039bc315c5d96758b8729123c61333)
BibTeX
@misc{dcf2722107fca3,
title = {Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents — arXiv:2608.14606},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14606},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f2722107fca3; sha256 1c8d7cdd010849b3af9cedf8f3594bee9d039bc315c5d96758b8729123c61333},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14606. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations mostly ask whether individual answers look plausible. We ask a stricter, psychometric question: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce an open Lithuanian organisational-psychology dataset (n=263 employees; three validated instruments; 68 items, 12 subscales) and condition 37 LLMs on real respondent profiles under a persona-disclosure ladder, presentation, reasoning-effort and cross-language ablations, counterfactual demographic swaps, and a memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five statistical baselines, a held-out human ceiling, and the real answers of the same respondents. LLMs reproduce the direction of human psychometric relationships, but statistical baselines with no language model match or beat them. The best LLM (PSS 0.81) is level with a Gaussian copula fit only on other respondents (0.82) and below the real answers of the same 100 people (0.89). A simple classifier separates every LLM sample from humans (median AUC 0.99), and verbatim recall of the items is rare (at most 4.7%) and unrelated to PSS. Counterfactual swaps move answers in the direction of human group differences and by a similar amount (role 0.49 and education 0.33 human SD, vs. 0.46 and 0.44 in humans). Gender swaps move them by 0.06 SD. Tucker's phi on UWES falls inside an item-permutation null for 10 of 37 models. Synthetic respondents compress the response scale (within-respondent SD 0.79 vs. 1.19), lose predictive validity on held-out humans (mean R^2=-0.14 vs. 0.28), and produce significant indirect effects on 4 of 7 mediation paths that are null in humans. LLM samples are therefore not a drop-in replacement for human survey data. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI). Categorie arXiv: Computers and Society (cs.CY) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Applications (stat.AP). Pagina del lavoro: https://arxiv.org/abs/2608.14606 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T16:29:13.908Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).