Metraipolis › archive › dc_e5c4abc79dad

Search-Based Testing of Modality Sensitivity in Vision Language Models — arXiv:2610.32360

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.32360
License (as archived)
CC0-1.0
Archived
2026-10-26T20:15:42.525Z
SHA-256 of the archived text
5183f22759ef14817ef8e286cc21578b5a26e325f8228d6cee3c1bf7ef9fc57f
Stable URL of this page
https://metraipolis.com/documento/dc_e5c4abc79dad

How to cite

Search-Based Testing of Modality Sensitivity in Vision Language Models — arXiv:2610.32360 — arXiv (en), https://arxiv.org/abs/2610.32360 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e5c4abc79dad (Metraipolis archive dc_e5c4abc79dad, sha256 5183f22759ef14817ef8e286cc21578b5a26e325f8228d6cee3c1bf7ef9fc57f)
BibTeX
@misc{dce5c4abc79dad,
  title = {Search-Based Testing of Modality Sensitivity in Vision Language Models — arXiv:2610.32360},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.32360},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e5c4abc79dad; sha256 5183f22759ef14817ef8e286cc21578b5a26e325f8228d6cee3c1bf7ef9fc57f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32360. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision-language models (VLMs) process images and text jointly, yet robustness evaluations often perturb these inputs separately or assess only predefined corruption levels. Such evaluations provide limited evidence of which modality is more susceptible to perturbations and whether smaller changes across both modalities can jointly expose failures. We present MoST, a black-box testing framework that characterizes modality sensitivity by comparing image-only, text-only, and joint perturbations. MoST uses multi-objective search to select and combine perturbations and adjust their severity, seeking prediction degradation while limiting changes to images and prompts. We evaluate MoST on four VLMs performing object localization on ImageNet and driving images. The models exhibit distinct sensitivities: textual perturbations dominate for some, whereas others are susceptible across both modalities. In several configurations, joint perturbations match or exceed the effectiveness of single-modality perturbations with smaller modifications to each input. In a human study, 60-76% of assessments across models judge the perturbed inputs sufficiently understandable to perform the task, and human localization is more accurate than that of the corresponding VLMs. These results show that evaluating modalities separately can overlook failure-inducing combinations and that modality sensitivity provides a more detailed account of VLM robustness than aggregate failure rates alone.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.32360 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:49:18.388Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).