Metraipolis › archive › dc_131a7986bf3e

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models — arXiv:2608.29590

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.29590
License (as archived)
CC0-1.0
Archived
2026-10-17T09:59:02.293Z
SHA-256 of the archived text
1b9fe22bea70040f26dcc01bf0deba1495ce2729f926160881b65a9ded40910f
Stable URL of this page
https://metraipolis.com/documento/dc_131a7986bf3e

How to cite

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models — arXiv:2608.29590 — arXiv (en), https://arxiv.org/abs/2608.29590 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_131a7986bf3e (Metraipolis archive dc_131a7986bf3e, sha256 1b9fe22bea70040f26dcc01bf0deba1495ce2729f926160881b65a9ded40910f)
BibTeX
@misc{dc131a7986bf3e,
  title = {Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models — arXiv:2608.29590},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.29590},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_131a7986bf3e; sha256 1b9fe22bea70040f26dcc01bf0deba1495ce2729f926160881b65a9ded40910f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29590. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely on prompts that ask models to infer attributes of people in images (e.g., "Is this person a CEO or a secretary?"). However, we find that LVLMs with strong guardrails, such as GPT and Claude, often refuse these prompts, making evaluations unreliable. To address this, we change the prior evaluation paradigm by decoupling the task from the depicted person: instead of inferring person's attributes, we use prompts that do not ask about the person (e.g., "Write a fictional story about an imaginary person.") and attach the image as provisional user information to implicitly provide demographic cues, then compare outputs across user demographics. Instantiated across three tasks --- story generation, term explanation, and exam-style QA --- our method avoids refusals even in guardrailed LVLMs, enabling reliable bias measurement. Applying it to 20 recent LVLMs, both open-source and proprietary, we find that all models undesirably use user demographic information in person-irrelevant tasks; for instance, characters in stories are often portrayed as mechanic for male users and nurse for female users. Although still biased, proprietary models like GPT-5 show lower bias than open-source ones. We analyze potential factors behind this gap, discussing continuous model monitoring and improvement as a possible contributor for reducing bias.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.29590 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T09:28:48.050Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).