Metraipolis › archive › dc_d479526bbada

Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models — arXiv:2610.40893

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40893
License (as archived): CC0-1.0
Archived: 2026-11-02T07:03:39.908Z
SHA-256 of the archived text: 5f8265e0281df80e016b962a482f747f67be310290496a483dc79ea1976d6776
Stable URL of this page: https://metraipolis.com/documento/dc_d479526bbada

How to cite

Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models — arXiv:2610.40893 — arXiv (en), https://arxiv.org/abs/2610.40893 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d479526bbada (Metraipolis archive dc_d479526bbada, sha256 5f8265e0281df80e016b962a482f747f67be310290496a483dc79ea1976d6776)

BibTeX

@misc{dcd479526bbada,
  title = {Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models — arXiv:2610.40893},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40893},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d479526bbada; sha256 5f8265e0281df80e016b962a482f747f67be310290496a483dc79ea1976d6776},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40893. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Fairness benchmarks for large language models (LLMs) are overwhelmingly English-born and American in their social categories; translated versions carry those categories into societies where they do not apply, and almost none fix their hypotheses before the models are run. We present Qistas (Arabic qistas, "the just balance"), a protocol for evaluating how LLMs treat the social, confessional, and regional constituencies of a society, portable across countries and instantiated first for Iraq. In its pilot form it measures only what needs no verdict on contested truth: procedural symmetry across matched counterfactual pairs, differential refusal, stereotype reliance under ambiguity, and terminological framing of documented events. Every measurement is paired by construction and analysed with paired machinery (sign-flip permutation tests, exact McNemar tests, template-clustered bootstrap intervals, Holm-corrected families), with a power analysis that states, before any data exist, the smallest effect each axis can detect after the design effect of template clustering. We release the authoring apparatus (a pruned 3x5 taxonomy with four cells closed by design, per-type checklists, a cumulative rulings log, a structural audit, and a human review notebook) and report its first field test: a semantic review of a 58-item seed bank in which 24 of 25 template groups failed at least one check. The defect patterns it exposed (evaluation criteria leaking into prompts, token-level swaps that break Arabic agreement, epistemically unbalanced pairs, single-theme refusal probes, unwritten dialect twins counted as coverage) are, we argue, general hazards of counterfactual fairness benchmarks. This is a protocol paper, not a results paper: no model has been evaluated, and the accompanying pre-registration commits the hypotheses, metrics, exclusion rules, and analysis plan that the pilot will follow.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2610.40893 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T06:57:13.205Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).