Metraipolis › archive › dc_139df73eb194
One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs — arXiv:2609.14860
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.14860
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T22:53:24.695Z
- SHA-256 of the archived text
d3d43fbf5bffa2bd16170bfdc3d128c10e544ed8ad56c8fa2f3bd1f9e2dc061a- Stable URL of this page
https://metraipolis.com/documento/dc_139df73eb194
How to cite
One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs — arXiv:2609.14860 — arXiv (en), https://arxiv.org/abs/2609.14860 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_139df73eb194 (Metraipolis archive dc_139df73eb194, sha256 d3d43fbf5bffa2bd16170bfdc3d128c10e544ed8ad56c8fa2f3bd1f9e2dc061a)
BibTeX
@misc{dc139df73eb194,
title = {One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs — arXiv:2609.14860},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.14860},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_139df73eb194; sha256 d3d43fbf5bffa2bd16170bfdc3d128c10e544ed8ad56c8fa2f3bd1f9e2dc061a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14860. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benchmarks such as BBQ have become the de facto standard for fairness evaluation across major model families. We argue that these benchmarks are too easy to support their role: training Qwen 2.5 7B Base with Group Relative Policy Optimization (GRPO) on a single BBQ example, or placing that example in context as a one-shot demonstration for in-context learning (ICL), lifts mean BBQ accuracy from 79.9% to 92.9% and 99.0%, respectively, closing 80% of the gap to its large-scale RLHF counterpart (96.1%) with GRPO, and surpassing it with ICL. These effects generalize across model families. A cross-conditioning analysis shows the improvement is carried by the reasoning traces generated by the model, and one example suffices to elicit a category-agnostic ``missing evidence'' reasoning pattern. We argue that BBQ-style multiple-choice abstention benchmarks measure a single structural cue, and a model that solves them does not thereby become fair. We call for evaluation suites that cover a broader spectrum of fairness alignment. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.14860 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:29:03.745Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).