Metraipolis › archive › dc_bf245c46bed0

Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines — arXiv:2610.41486

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41486
License (as archived)
CC0-1.0
Archived
2026-11-03T07:11:32.770Z
SHA-256 of the archived text
fd04cc08d6ae65b96d536af86780bb147d1ab971f7db6117618d4c744ed99649
Stable URL of this page
https://metraipolis.com/documento/dc_bf245c46bed0

How to cite

Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines — arXiv:2610.41486 — arXiv (en), https://arxiv.org/abs/2610.41486 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf245c46bed0 (Metraipolis archive dc_bf245c46bed0, sha256 fd04cc08d6ae65b96d536af86780bb147d1ab971f7db6117618d4c744ed99649)
BibTeX
@misc{dcbf245c46bed0,
  title = {Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines — arXiv:2610.41486},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41486},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf245c46bed0; sha256 fd04cc08d6ae65b96d536af86780bb147d1ab971f7db6117618d4c744ed99649},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41486. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41486 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:09:31.774Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).