Metraipolis › archive › dc_4605af075d73

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models — arXiv:2609.22234

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.22234
License (as archived)
CC0-1.0
Archived
2026-10-09T06:49:22.409Z
SHA-256 of the archived text
cb336654e647ceea6dd2ba1a15e095bfddfdec206698bdc22e6085c8330b3cb6
Stable URL of this page
https://metraipolis.com/documento/dc_4605af075d73

How to cite

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models — arXiv:2609.22234 — arXiv (en), https://arxiv.org/abs/2609.22234 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4605af075d73 (Metraipolis archive dc_4605af075d73, sha256 cb336654e647ceea6dd2ba1a15e095bfddfdec206698bdc22e6085c8330b3cb6)
BibTeX
@misc{dc4605af075d73,
  title = {Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models — arXiv:2609.22234},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22234},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4605af075d73; sha256 cb336654e647ceea6dd2ba1a15e095bfddfdec206698bdc22e6085c8330b3cb6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22234. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed, or encountered during agentic tasks. Positing multimodal IH alignment as a reasoning problem, we train VLMs using reinforcement learning with rule-based rewards, comparing text-only, image-only, and mixed-modality supervision. We find that text-only IH training partially transfers to multimodal attacks, failing when models must decode, reconstruct, or reason over instructions across modalities. Image-based training improves robustness beyond text-only supervision, while mixed-modality training performs best overall. Importantly, the benefits generalize beyond the synthetic typographic training setting to real-image and web-agent safety tasks, while largely preserving general multimodal capability, showing that lightweight, verifiable supervision can meaningfully improve VLM robustness under adversarial, cross-modal, and interactive instruction conflicts.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.22234 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:44.907Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).