Metraipolis › archive › dc_11366d1bd929

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP — arXiv:2609.01103

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01103
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:28.998Z
SHA-256 of the archived text
bf1c7b7e8fa9968f085b5a274b7c4aabe9f3b076c8ccf857cd6146368e24c108
Stable URL of this page
https://metraipolis.com/documento/dc_11366d1bd929

How to cite

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP — arXiv:2609.01103 — arXiv (en), https://arxiv.org/abs/2609.01103 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_11366d1bd929 (Metraipolis archive dc_11366d1bd929, sha256 bf1c7b7e8fa9968f085b5a274b7c4aabe9f3b076c8ccf857cd6146368e24c108)
BibTeX
@misc{dc11366d1bd929,
  title = {When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP — arXiv:2609.01103},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01103},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_11366d1bd929; sha256 bf1c7b7e8fa9968f085b5a274b7c4aabe9f3b076c8ccf857cd6146368e24c108},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01103. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reducing the modality gap between image and text representations in CLIP is widely expected to improve cross-modal alignment and downstream performance. However, a smaller average image-text gap does not necessarily lead to consistent accuracy gains. We analyze this mismatch from the perspective of the decision structure in zero-shot classification, i.e. selecting the most similar class-text prototype for an input image. Zero-shot accuracy depends not only on average image--text alignment, but also on class-wise decision margins. Using Linear correction as an analytically tractable case, we show that modality gap correction can alter the relative decision structure among classes and cause predictions to concentrate on a small subset of classes. We refer to this output-space failure mode as prediction-level hubness. Furthermore, experiments across multiple datasets show that accuracy degradation under gap correction is consistently associated with increased prediction concentration, both for Linear correction and for learning-based correction methods. This provides a systematic explanation of why modality gap reduction does not consistently improve CLIP zero-shot accuracy from the perspective of downstream decision structure. Our results suggest that gap correction should be evaluated not only by average alignment, but also by its impact on downstream prediction structure.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.01103 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.843Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).