Metraipolis › archive › dc_4605af075d73
Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models — arXiv:2609.22234
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.22234
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:49:22.409Z
- SHA-256 of the archived text
cb336654e647ceea6dd2ba1a15e095bfddfdec206698bdc22e6085c8330b3cb6- Stable URL of this page
https://metraipolis.com/documento/dc_4605af075d73
How to cite
Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models — arXiv:2609.22234 — arXiv (en), https://arxiv.org/abs/2609.22234 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4605af075d73 (Metraipolis archive dc_4605af075d73, sha256 cb336654e647ceea6dd2ba1a15e095bfddfdec206698bdc22e6085c8330b3cb6)
BibTeX
@misc{dc4605af075d73,
title = {Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models — arXiv:2609.22234},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.22234},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4605af075d73; sha256 cb336654e647ceea6dd2ba1a15e095bfddfdec206698bdc22e6085c8330b3cb6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22234. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed, or encountered during agentic tasks. Positing multimodal IH alignment as a reasoning problem, we train VLMs using reinforcement learning with rule-based rewards, comparing text-only, image-only, and mixed-modality supervision. We find that text-only IH training partially transfers to multimodal attacks, failing when models must decode, reconstruct, or reason over instructions across modalities. Image-based training improves robustness beyond text-only supervision, while mixed-modality training performs best overall. Importantly, the benefits generalize beyond the synthetic typographic training setting to real-image and web-agent safety tasks, while largely preserving general multimodal capability, showing that lightweight, verifiable supervision can meaningfully improve VLM robustness under adversarial, cross-modal, and interactive instruction conflicts. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.22234 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T06:48:44.907Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).