Metraipolis › archive › dc_cdf569fddcc9

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder — arXiv:2610.11374

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11374
License (as archived)
CC0-1.0
Archived
2026-10-09T05:57:37.893Z
SHA-256 of the archived text
666f1fbc08db9522ba90901def6b71875df494dceae77f2994c6e4ebac99316c
Stable URL of this page
https://metraipolis.com/documento/dc_cdf569fddcc9

How to cite

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder — arXiv:2610.11374 — arXiv (en), https://arxiv.org/abs/2610.11374 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cdf569fddcc9 (Metraipolis archive dc_cdf569fddcc9, sha256 666f1fbc08db9522ba90901def6b71875df494dceae77f2994c6e4ebac99316c)
BibTeX
@misc{dccdf569fddcc9,
  title = {Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder — arXiv:2610.11374},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11374},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cdf569fddcc9; sha256 666f1fbc08db9522ba90901def6b71875df494dceae77f2994c6e4ebac99316c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11374. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

CLIP serves as a foundational vision-language model and the de facto vision encoder for downstream VLMs such as LLaVA. Post-training offers a lightweight route to refine CLIP, but recent work argues that the standard contrastive loss is unsuitable for post-training due to catastrophic forgetting under small batches, motivating designs that abandon the contrastive objective in favor of distillation. We revisit this premise and find that, for the InfoNCE objective, the reported forgetting is driven primarily not by insufficient negatives but by an inappropriate magnitude of the contrastive temperature $\tau$: with $\tau$ set sufficiently small, contrastive post-training improves rather than degrades the pretrained CLIP, which we explain through the temperature dependence of the InfoNCE gradient. Building on this finding, we propose \textbf{ComCLIP}, a lightweight single-epoch post-training recipe that freezes CLIP's text encoder---so the refined vision encoder is a drop-in replacement with unchanged architecture and inference cost---and trains the vision encoder with a properly-tempered contrastive loss, an MSE anchoring loss against the original CLIP, and a relational distillation loss from DINOv2. Over multiple seeds, ComCLIP matches the self-distillation baseline CLIP-Refine on zero-shot classification while significantly improving the transferability of visual features, measured by linear probing ($48.99$ vs.\ $42.28$ on ViT-B/16), and on ViT-L/14 it also improves MMVP over CLIP-Refine ($24.20$ vs.\ $19.01$); CLIP-Refine remains stronger on image-text retrieval. Used as a drop-in vision encoder for LLaVA-1.5-7B without re-aligning the projector or LLM, ComCLIP yields no net change across $8$ VLM benchmarks, i.e., the refinement does not break downstream compatibility. Code and models are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.11374 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:01.612Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).