Metraipolis › archive › dc_35263f38fcd6
Do Language Models Understand Igbo Culture? Beyond Multiple Choice — arXiv:2610.35168
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.35168
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T08:12:18.254Z
- SHA-256 of the archived text
c5dbd46cebba3b112ef5a6d9c6f75d59954c7126a10c0d65b2d2c3f2472f4074- Stable URL of this page
https://metraipolis.com/documento/dc_35263f38fcd6
How to cite
Do Language Models Understand Igbo Culture? Beyond Multiple Choice — arXiv:2610.35168 — arXiv (en), https://arxiv.org/abs/2610.35168 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_35263f38fcd6 (Metraipolis archive dc_35263f38fcd6, sha256 c5dbd46cebba3b112ef5a6d9c6f75d59954c7126a10c0d65b2d2c3f2472f4074)
BibTeX
@misc{dc35263f38fcd6,
title = {Do Language Models Understand Igbo Culture? Beyond Multiple Choice — arXiv:2610.35168},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.35168},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_35263f38fcd6; sha256 c5dbd46cebba3b112ef5a6d9c6f75d59954c7126a10c0d65b2d2c3f2472f4074},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.35168. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use the Igbo language as a case study. Existing benchmarks test translated Western ideas, lacking depth. To address this, we introduce CAB-Igbo, a benchmark developed from indigenous cultural sources and verified by native speakers across six domains. We evaluated diverse models under varying conditions, incorporating regional personas. Performance was assessed using multiple-choice accuracy and human evaluations across Epistemic Fidelity, Representational Richness, Linguistic Quality, and Pragmatic Proficiency. Results reveal a stark dissociation between language fluency and cultural knowledge. Through statistical testing, we show that for most models, the severe lack of Epistemic Fidelity and Representational Richness is already present in English and does not significantly worsen when switching to Igbo. Instead, native language prompting mainly degrades Linguistic Quality and Pragmatic Proficiency. Gemini 2.0 Flash is the only model whose cultural knowledge scores improve under Igbo prompting. We find that current models struggle to demonstrate the knowledge needed for true alignment with Igbo, and multiple-choice formats allow pattern-matching to appear as understanding. Furthermore, automated judges agree with humans on linguistic quality but fail on epistemic fidelity. Ultimately, cultural evaluation must be built from community knowledge, establishing human evaluation as the gold standard to prevent misrepresenting low-resource communities globally. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.35168 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-27T07:56:00.255Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).