Metraipolis › archive › dc_2b81811cab8a
Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict — arXiv:2609.00550
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.00550
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:41:17.474Z
- SHA-256 of the archived text
b017d9f6ca0a7cbf2623d7f6d348e7fd1c5156aa4f7a92cbcdcd80cfdb7ba698- Stable URL of this page
https://metraipolis.com/documento/dc_2b81811cab8a
How to cite
Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict — arXiv:2609.00550 — arXiv (en), https://arxiv.org/abs/2609.00550 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2b81811cab8a (Metraipolis archive dc_2b81811cab8a, sha256 b017d9f6ca0a7cbf2623d7f6d348e7fd1c5156aa4f7a92cbcdcd80cfdb7ba698)
BibTeX
@misc{dc2b81811cab8a,
title = {Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict — arXiv:2609.00550},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.00550},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2b81811cab8a; sha256 b017d9f6ca0a7cbf2623d7f6d348e7fd1c5156aa4f7a92cbcdcd80cfdb7ba698},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00550. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multimodal large language models (MLLMs) are increasingly provided with contextual evidence in heterogeneous forms: as a text passage, as a rendered image of the same passage, or as both together. However, it remains unclear how consistently these surface forms are processed, especially when the evidence conflicts with the model's parametric knowledge. We study modality robustness under knowledge conflict across 13 MLLMs and two datasets, and find them far from robust. (1) Contrary to common belief, models favor a context that contradicts parametric knowledge more readily in image form than in text form; (2) when a contradicting text and image are presented together, the preferred modality is essentially arbitrary, varying with input order, model, and dataset. We further demonstrate that this instability has practical consequences: it degrades performance in multimodal RAG and can be exploited by adversarial attacks. To alleviate this brittleness, we examine several simple techniques---prompting, steering, supervised fine-tuning (SFT), and direct preference optimization; the majority prove ineffective, whereas SFT achieves moderate success. We therefore call for greater awareness of this inconsistency and argue that it is fundamental, demanding attention at multiple training stages. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.00550 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:33:38.813Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).