Metraipolis › archive › dc_ddb607fc99d9

Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines — arXiv:2610.41486

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41486
License (as archived)
CC0-1.0
Archived
2026-11-03T07:51:21.932Z
SHA-256 of the archived text
72cf9f84466ed6d5d8b64d0c35d871170fdb13ea731bbd52dc64931f9c1f7a02
Stable URL of this page
https://metraipolis.com/documento/dc_ddb607fc99d9

How to cite

Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines — arXiv:2610.41486 — arXiv (en), https://arxiv.org/abs/2610.41486 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ddb607fc99d9 (Metraipolis archive dc_ddb607fc99d9, sha256 72cf9f84466ed6d5d8b64d0c35d871170fdb13ea731bbd52dc64931f9c1f7a02)
BibTeX
@misc{dcddb607fc99d9,
  title = {Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines — arXiv:2610.41486},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41486},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ddb607fc99d9; sha256 72cf9f84466ed6d5d8b64d0c35d871170fdb13ea731bbd52dc64931f9c1f7a02},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41486. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41486 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:49:31.066Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).