Metraipolis › archive › dc_ccffaf1f2d4a

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models — arXiv:2609.16006

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.16006
License (as archived)
CC0-1.0
Archived
2026-10-15T19:23:21.765Z
SHA-256 of the archived text
0a9968ba7a3a23a76889192bb099e965cc73a93df4a7ee73c46ba1478e4f92f1
Stable URL of this page
https://metraipolis.com/documento/dc_ccffaf1f2d4a

How to cite

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models — arXiv:2609.16006 — arXiv (en), https://arxiv.org/abs/2609.16006 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ccffaf1f2d4a (Metraipolis archive dc_ccffaf1f2d4a, sha256 0a9968ba7a3a23a76889192bb099e965cc73a93df4a7ee73c46ba1478e4f92f1)
BibTeX
@misc{dcccffaf1f2d4a,
  title = {Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models — arXiv:2609.16006},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.16006},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ccffaf1f2d4a; sha256 0a9968ba7a3a23a76889192bb099e965cc73a93df4a7ee73c46ba1478e4f92f1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16006. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test what a model knows rather than how it behaves when giving open-ended recommendations, opinions, and guidance. We introduce AraBehave: 1,623 culturally grounded, open-ended Arabic prompts with 29,214 cultural-appropriateness judgments from native speakers across several Arab regions, plus a scoring model whose predictions correlate strongly with human judgments on unseen systems (Pearson r=0.74). Evaluating three Arabic-centric and three frontier LLMs, we find that cultural appropriateness is not a single capability but decomposes into two largely independent components: normative stance and grounded cultural accuracy. The best general-purpose and best Arabic-centric models score identically (3.84 vs. 3.83 of 5) yet almost never fail for the same reason: general-purpose models exhibit strong factual grounding but a culturally inappropriate normative stance, being penalized for secular framing and false balance on culturally settled matters (28--33% of their low-score rationales), while the best Arabic-centric model adopts the expected stance but is penalized for fabricated hadith and misquoted verses (29%). Stance is cheap and fragile: one sentence of cultural instruction lifts Gemini to 4.57, above every Arabic-specialized model. Conversely, a generic ``answer clearly and objectively'' prompt costs Allam-7B 0.68 points, while asking the same questions in English lowers scores for every model but one. Grounding instead tracks scale and Arabic alignment data, and disappears when culturally aware instruction tuning is replaced by a culture-neutral corpus. General safety benchmarks see none of this: they saturate above 89 while cultural scores span 2.71-3.84. We will release the benchmark, annotations, and the scoring model.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.CL).
Categorie arXiv: Computers and Society (cs.CY) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.16006 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:32:32.879Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).