Metraipolis › archive › dc_b620f97be7bf

The Uncontrolled Variable: Vision-Language Refusal Is Conditioned on the Image-Attachment Interface, and Not Robust to Irrelevant Image Properties — arXiv:2609.26174

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.26174
License (as archived)
CC0-1.0
Archived
2026-10-08T13:46:58.171Z
SHA-256 of the archived text
fe2aa536f426c912dc0a9a822d7f3770887ab3922332a5995373621ad0a95707
Stable URL of this page
https://metraipolis.com/documento/dc_b620f97be7bf

How to cite

The Uncontrolled Variable: Vision-Language Refusal Is Conditioned on the Image-Attachment Interface, and Not Robust to Irrelevant Image Properties — arXiv:2609.26174 — arXiv (en), https://arxiv.org/abs/2609.26174 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b620f97be7bf (Metraipolis archive dc_b620f97be7bf, sha256 fe2aa536f426c912dc0a9a822d7f3770887ab3922332a5995373621ad0a95707)
BibTeX
@misc{dcb620f97be7bf,
  title = {The Uncontrolled Variable: Vision-Language Refusal Is Conditioned on the Image-Attachment Interface, and Not Robust to Irrelevant Image Properties — arXiv:2609.26174},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.26174},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b620f97be7bf; sha256 fe2aa536f426c912dc0a9a822d7f3770887ab3922332a5995373621ad0a95707},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26174. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We show that aligned vision-language models also condition refusal on a property of a request's form: whether an image is attached, holding everything the request asks fixed. Attaching a blank canvas, an image that cannot be read, cannot relate to the request, and is byte-identical across every prompt in its condition, shifts benign refusal by tens of points. The shift is not blanket caution but a threshold shift: genuinely neutral instructions are almost unaffected (<=2 percentage points on three of four hosted models) while borderline-benign prompts move +23 to +51 points, so the cost falls on sensitivity-adjacent traffic, meaning benign questions about privacy, self-harm, violence and illegal activity. There is a benign reading of such a threshold, namely that attachment correlates with risk in real traffic, and we take it seriously; a black-box study cannot measure that correlation and we do not claim to. What it can test is whether the response to attachment is robust to variation carrying no information about the request, and on four independent measurements it is not. It varies with canvas colour and pixel count. Its sign inverts across checkpoints. It survives an explicit instruction to disregard the image. And on one model it fires on a bare assertion that an attachment exists, with nothing attached and the modality word contributing none of it. Finally we price it. On a matched harmful set the same canvas does lower attack success, so the cue buys something. But the charge is decoupled from the purchase: the checkpoint with the least harmful headroom we measure, completing only 2% of plain harmful requests, still pays the benign cost in full, and across our models the harmful-side denominator falls as alignment improves while the benign cost does not track it down. Image presence is not a conservative default that a deployer chose and priced. It is an uncontrolled variable.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AI).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.26174 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:18:38.289Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).