Metraipolis › archive › dc_ae4e09ff871c

What Does a Safety Label Measure? Auditing Situational-Safety Benchmarks for Multimodal LLMs — arXiv:2611.01125

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.01125
License (as archived): CC0-1.0
Archived: 2026-11-03T10:49:29.977Z
SHA-256 of the archived text: 361b62412fd9fef978d84d32e33c7b44b5fafd528c04184222bf2b0d0f0ec5d0
Stable URL of this page: https://metraipolis.com/documento/dc_ae4e09ff871c

How to cite

What Does a Safety Label Measure? Auditing Situational-Safety Benchmarks for Multimodal LLMs — arXiv:2611.01125 — arXiv (en), https://arxiv.org/abs/2611.01125 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ae4e09ff871c (Metraipolis archive dc_ae4e09ff871c, sha256 361b62412fd9fef978d84d32e33c7b44b5fafd528c04184222bf2b0d0f0ec5d0)

BibTeX

@misc{dcae4e09ff871c,
  title = {What Does a Safety Label Measure? Auditing Situational-Safety Benchmarks for Multimodal LLMs — arXiv:2611.01125},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01125},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ae4e09ff871c; sha256 361b62412fd9fef978d84d32e33c7b44b5fafd528c04184222bf2b0d0f0ec5d0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01125. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Situational-safety benchmarks label each scene SAFE or UNSAFE and score a multimodal large language model (MLLM) against that label, usually by judging its free-form answer, and these scores guide deployment. We ask what such a label measures. A label score can come from the scene, the request or the benchmark's own files. We propose the Benchmark Card, seven measured fields that tell them apart, and fill it for six benchmarks. Only MSSBench chat leaks the label through its image files: image size alone reaches 82.3% accuracy, matching or beating six of seven commercial models. Elsewhere the signal outside the scene comes from design. In MOSSBench the item partition fixes the label, and on HoliSafe the request alone ranks pairs about as well as the scene. On our corrected chat split, the size rule falls to chance. We then introduce the Spot-Stop decomposition, which compares ranking with labelling on pairs of a safe and an unsafe scene. It splits the gap above a label-swap null into three parts: a threshold part, overlap between safe and unsafe scores, and order that holds only within a pair. Most of fourteen open MLLMs rank better than they label, beyond what their AUROC predicts. On chat, one shared threshold repairs a third to a half of the excess, and label-free recentering recovers it on balanced batches. Recentering on unlabelled scenes of the same situation improves ranking on VSCBench for 13 of 14 models.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.MA, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2611.01125 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:22.881Z dalla pagina https://arxiv.org/list/cs.MA/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).