Metraipolis › archive › dc_a214c2491027
Measurement Validity in LLM Cultural Alignment — arXiv:2608.29266
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.29266
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:47:36.621Z
- SHA-256 of the archived text
ce3d097c465559335b7dc42dca2fb989929f9fbb6ed705534ec62752386dc6ee- Stable URL of this page
https://metraipolis.com/documento/dc_a214c2491027
How to cite
Measurement Validity in LLM Cultural Alignment — arXiv:2608.29266 — arXiv (en), https://arxiv.org/abs/2608.29266 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a214c2491027 (Metraipolis archive dc_a214c2491027, sha256 ce3d097c465559335b7dc42dca2fb989929f9fbb6ed705534ec62752386dc6ee)
BibTeX
@misc{dca214c2491027,
title = {Measurement Validity in LLM Cultural Alignment — arXiv:2608.29266},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.29266},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a214c2491027; sha256 ce3d097c465559335b7dc42dca2fb989929f9fbb6ed705534ec62752386dc6ee},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29266. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Researchers increasingly treat LLM survey responses as a proxy for human cultural values. This includes projecting model outputs onto instruments like the Inglehart-Welzel Cultural Map and drawing conclusions about which cultures a model resembles. While a model's answer to a value-laden questions may be interpreted as a cultural signal, it also carries sampling noise and, can be quite sensitive to question framing. In this paper, we separate survey responses, sampling noise and question framing for multiple LLMs. We decompose response variance from these models into variation across random seeds, prompt rewordings. We employ noise-to-signal ratio (NSR) to test whether a model's apparent cultural position is distinguishable from noise. When applied across a dozen models from four geographic origins, calibrated against 88 Integrated Values Survey countries, the answer is often no. NSR exceeds 1.0 on 49 of 117 valid model-question pairs (42%), reaching 5.56 in the worst case. Two models even refuse to answer sufficient number of survey questions outright. Our results corroborate previous findings that LLMs cluster toward Western, English-speaking cultural positions. However, what does not hold up in this study is the precision with which anyone can currently interpret a specific model's coordinates: prompt tone alone can shift a model by 2.4 map units, comparable to the distance between actual countries in the Inglehart-Welzel Cultural Map. These findings suggest that cultural attribution from LLM survey responses requires establishing the reliability of the underlying measurements before interpreting model coordinates as evidence of cultural representation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Physics and Society (physics.soc-ph) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.29266 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.736Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).