Metraipolis › archive › dc_e5e50e8a4ace
Bias Audits Detect Bias but Disagree on Ranking: Evidence from Ten Instruments and Ten Frontier Models — arXiv:2609.15995
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15995
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T19:17:04.135Z
- SHA-256 of the archived text
6d2de01581679b57e8a6eb2c0dfeace7e3f1f7db0a7e2e282d3aea34b1808196- Stable URL of this page
https://metraipolis.com/documento/dc_e5e50e8a4ace
How to cite
Bias Audits Detect Bias but Disagree on Ranking: Evidence from Ten Instruments and Ten Frontier Models — arXiv:2609.15995 — arXiv (en), https://arxiv.org/abs/2609.15995 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e5e50e8a4ace (Metraipolis archive dc_e5e50e8a4ace, sha256 6d2de01581679b57e8a6eb2c0dfeace7e3f1f7db0a7e2e282d3aea34b1808196)
BibTeX
@misc{dce5e50e8a4ace,
title = {Bias Audits Detect Bias but Disagree on Ranking: Evidence from Ten Instruments and Ten Frontier Models — arXiv:2609.15995},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15995},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e5e50e8a4ace; sha256 6d2de01581679b57e8a6eb2c0dfeace7e3f1f7db0a7e2e282d3aea34b1808196},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15995. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Emerging AI regulation mandates bias audits of high-risk systems, and audit scores are beginning to be used to rank models. Both uses assume different audit tools measure the same thing well enough to compare. We test that assumption directly, running ten extrinsic audit instruments over a shared panel of ten frontier models through one pooled inference gateway, first on occupational gender bias, then on age and socioeconomic status. Detection succeeds while ranking fails. Eight of ten tools detect bias with confidence intervals clear of zero; two widely cited direct-probe benchmarks are saturated because frontier models now answer neutrally. But cross-tool rank agreement is indistinguishable from chance (Kendall's W=0.07, p=0.83). A positive control with six deliberately weaker models separates two explanations: within-tool reliability recovers once the panel spans real capability gaps, yet cross-tool ranking never recovers, which points to the tools measuring different constructs rather than one construct noisily. Even the direction of bias splits by audit format: forced-choice decision tools mostly over-correct (toward women, and toward working-class candidates in 273 of 278 hiring decisions), while free generation and default coreference stay stereotype-congruent. The pattern replicates on socioeconomic status; an apparent ranking agreement on age dissolves under the paper's own tool-inclusion rules. The practical message: a single audit can detect bias and estimate its direction within its own operationalization, but no single audit supports ranking one model against another. All raw responses, code, and the analysis that recomputes every reported number from source are available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.15995 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T18:32:32.778Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).