Metraipolis › archive › dc_5c5b2f6cbf92

Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models — arXiv:2611.09897

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.09897
License (as archived)
CC0-1.0
Archived
2026-11-10T18:44:57.665Z
SHA-256 of the archived text
d9dea81a21429b636b2b23f8d4dcfd8a03d6cca04b53e9220d032068dd066ae9
Stable URL of this page
https://metraipolis.com/documento/dc_5c5b2f6cbf92

How to cite

Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models — arXiv:2611.09897 — arXiv (en), https://arxiv.org/abs/2611.09897 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5c5b2f6cbf92 (Metraipolis archive dc_5c5b2f6cbf92, sha256 d9dea81a21429b636b2b23f8d4dcfd8a03d6cca04b53e9220d032068dd066ae9)
BibTeX
@misc{dc5c5b2f6cbf92,
  title = {Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models — arXiv:2611.09897},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.09897},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5c5b2f6cbf92; sha256 d9dea81a21429b636b2b23f8d4dcfd8a03d6cca04b53e9220d032068dd066ae9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09897. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Understanding the origins of the modality gap in vision language models remains a significant challenge. This work investigates whether differences in information density between modality specific tokens at the embedding layer contribute to this imbalance. We show that text tokens enter the model with much lower semantic density than visual tokens, driving early layers to focus on processing text representations before combining them with vision. Through mechanistic analysis, we demonstrate that replacing vision token positions with their semantically closest text token counterparts elicits increased attention to those positions in early layers. These findings provide a foundation for addressing the modality gap and provide insight to improve multimodal reasoning in VLM architectures.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.09897 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:52.463Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).