Metraipolis › archive › dc_9b00124a48fb

Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning — arXiv:2611.00608

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00608
License (as archived)
CC0-1.0
Archived
2026-11-03T18:16:13.544Z
SHA-256 of the archived text
01d27292d425c5bbe3838b42d3209c809681602b4d4eb4788cb5bfb93365394b
Stable URL of this page
https://metraipolis.com/documento/dc_9b00124a48fb

How to cite

Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning — arXiv:2611.00608 — arXiv (en), https://arxiv.org/abs/2611.00608 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9b00124a48fb (Metraipolis archive dc_9b00124a48fb, sha256 01d27292d425c5bbe3838b42d3209c809681602b4d4eb4788cb5bfb93365394b)
BibTeX
@misc{dc9b00124a48fb,
  title = {Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning — arXiv:2611.00608},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00608},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9b00124a48fb; sha256 01d27292d425c5bbe3838b42d3209c809681602b4d4eb4788cb5bfb93365394b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00608. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision-language model (VLM) unlearning is often treated as an objective-design problem: given multimodal forget data, the goal is to optimize a loss that removes undesirable behavior while preserving retained capabilities. We argue that this view overlooks an equally important data-centric question: how should each forget example be represented before optimization? Existing methods typically keep image-conditioned failures in their original image-text form, implicitly assuming that pixels are the right route for unlearning whenever pixels appear in the input. We challenge this assumption by constructing modality-decomposed versions of VLM benchmarks, enabling controlled comparisons among image-only, text-only, and multimodal representations of the same examples. Our modality-mixing experiments show that unsafe instruction-following behavior is more effectively suppressed through textual representations than through images alone, while purely text-only unlearning can still weaken visual grounding when the target behavior depends on visual evidence. Motivated by this trade-off, we introduce ModRoute, a gradient-guided per-sample modality routing strategy that selectively converts high visual-pressure examples to text-only form while keeping the remaining examples multimodal. Notably, on VLGuard with LLaVA-1.5-7B, ModRoute has the strongest composite unlearning-utility score of 0.847 at $\gamma=0.5$, a 2.5% improvement over the best random-switching score and $\geq$46% better than pure multimodal or text unlearning. Overall, our results show that effective VLM unlearning depends not only on the objective, but also on the composition and per-sample modality routing of the unlearning data.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.00608 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.527Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).