Metraipolis › archive › dc_23d8a12031a4

BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs — arXiv:2611.12407

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.12407
License (as archived)
CC0-1.0
Archived
2026-11-10T18:44:50.518Z
SHA-256 of the archived text
6eb8f13625cf0c4991648ac6cdef0b524262d4bfe1a5cad63c60ad881bd231f6
Stable URL of this page
https://metraipolis.com/documento/dc_23d8a12031a4

How to cite

BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs — arXiv:2611.12407 — arXiv (en), https://arxiv.org/abs/2611.12407 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_23d8a12031a4 (Metraipolis archive dc_23d8a12031a4, sha256 6eb8f13625cf0c4991648ac6cdef0b524262d4bfe1a5cad63c60ad881bd231f6)
BibTeX
@misc{dc23d8a12031a4,
  title = {BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs — arXiv:2611.12407},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.12407},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_23d8a12031a4; sha256 6eb8f13625cf0c4991648ac6cdef0b524262d4bfe1a5cad63c60ad881bd231f6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.12407. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Despite growing attention to social bias in large language models (LLMs), existing bias benchmarks provide limited coverage of culturally grounded evaluation in Bengali, particularly within Bangladesh's sociocultural context. To address this gap, we develop a bilingual extension of BanglaBBQ comprising 61,312 question-answering instances across 12 social categories in English and Bengali. The benchmark incorporates balanced answer permutations, contrasting question framings, and both ambiguous and disambiguated contexts to systematically examine model performance and stereotype-aligned bias across languages. We validate the dataset through structural checks, 220 statistical tests, and expert annotation of 150 instances by four annotators, achieving strong inter-annotator agreement (Fleiss' \k{appa}=0.8349). We evaluate eight LLMs across 490,496 instances and identify five key findings. First, disambiguated contexts yield substantially greater accuracy improvements in Bengali (+36.59 percentage points) than in English (+17.71). Second, all evaluated models exhibit lower accuracy in Bengali, although Mistral-Nemo shows the smallest cross-lingual accuracy gap (5.12 percentage points). Third, stereotype-aligned bias exhibits language and model dependent variation, increasing in Bengali for Gemma 2 9B but decreasing for Mistral-Nemo. Fourth, higher accuracy does not necessarily correspond to lower stereotype-aligned bias. Finally, bias varies considerably across social categories, with Qwen 2.5 7B exhibiting stronger stereotype alignment in Bengali for Politics and Physical Appearance. These findings demonstrate that cross-lingual performance and social bias are distinct dimensions of LLM behavior and underscore the importance of culturally grounded, language-sensitive benchmarks for evaluating fairness beyond English-centric settings.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.12407 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:52.454Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).