Metraipolis › archive › dc_c0fe7446f69c

NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria — arXiv:2611.06504

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.06504
License (as archived)
CC0-1.0
Archived
2026-11-05T18:17:59.840Z
SHA-256 of the archived text
5cfc522b71baf7622de4731fbf811dff46bcb70e29a42ab89c0918044d5091be
Stable URL of this page
https://metraipolis.com/documento/dc_c0fe7446f69c

How to cite

NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria — arXiv:2611.06504 — arXiv (en), https://arxiv.org/abs/2611.06504 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c0fe7446f69c (Metraipolis archive dc_c0fe7446f69c, sha256 5cfc522b71baf7622de4731fbf811dff46bcb70e29a42ab89c0918044d5091be)
BibTeX
@misc{dcc0fe7446f69c,
  title = {NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria — arXiv:2611.06504},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06504},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c0fe7446f69c; sha256 5cfc522b71baf7622de4731fbf811dff46bcb70e29a42ab89c0918044d5091be},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06504. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \naija, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \yoruba, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\% for the US/UK but 20\% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to \yoruba{} (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66--79\% on the UK and the US but none exceed 31\% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\% on culture and 98\% on politics against 26\% on statistics, so the gap is not specific to Nigeria.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.06504 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:12:49.060Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).