Metraipolis › archive › dc_2b4eea6e70fa
Visual Attention Faithfulness in Vision-Language Models is Heterogeneous — arXiv:2609.00830
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.00830
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:33:41.287Z
- SHA-256 of the archived text
53885e4bcca246321bbc988f765b0bdc7a27bb466e30fa088e94facdeb877c9c- Stable URL of this page
https://metraipolis.com/documento/dc_2b4eea6e70fa
How to cite
Visual Attention Faithfulness in Vision-Language Models is Heterogeneous — arXiv:2609.00830 — arXiv (en), https://arxiv.org/abs/2609.00830 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2b4eea6e70fa (Metraipolis archive dc_2b4eea6e70fa, sha256 53885e4bcca246321bbc988f765b0bdc7a27bb466e30fa088e94facdeb877c9c)
BibTeX
@misc{dc2b4eea6e70fa,
title = {Visual Attention Faithfulness in Vision-Language Models is Heterogeneous — arXiv:2609.00830},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.00830},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2b4eea6e70fa; sha256 53885e4bcca246321bbc988f765b0bdc7a27bb466e30fa088e94facdeb877c9c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00830. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Language Models (VLMs). We address this gap through causal perturbation analysis on current VLMs, evaluating both the comprehensiveness and sufficiency gap of attention-ranked visual tokens. Our analysis reveals that visual attention faithfulness is heterogeneous, manifesting in three distinct processing modes: Faithful-Sufficient, where top-$k$ attention tokens are both necessary and sufficient for prediction; Faithful-Distributed, where they are necessary but broader visual context remains required; and Non-Focal, where no localized attention region is individually necessary while visual information remains an essential trigger for prediction. Furthermore, human-annotated ground-truth regions satisfy comprehensiveness in only $\sim 60$% of cases compared with model attention rankings, revealing systematic divergence between model visual reliance and human intuition. We demonstrate these patterns across both general VQA on VQAv2 and document tasks on VRDU and ChartQA, showing that visual attention faithfulness varies systematically with processing demands and model architectures rather than being uniformly faithful or unfaithful. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.00830 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:15:12.826Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).