Metraipolis › archive › dc_e68411fd7f3f

Kernel-Based Steering of CLIP with Vision-Language Model Preferences — arXiv:2609.32203

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32203
License (as archived)
CC0-1.0
Archived
2026-10-07T22:39:36.367Z
SHA-256 of the archived text
dd36f2170b1787e416c0d6b42e689feb9553333ead7878b6c08ed213815a553f
Stable URL of this page
https://metraipolis.com/documento/dc_e68411fd7f3f

How to cite

Kernel-Based Steering of CLIP with Vision-Language Model Preferences — arXiv:2609.32203 — arXiv (en), https://arxiv.org/abs/2609.32203 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e68411fd7f3f (Metraipolis archive dc_e68411fd7f3f, sha256 dd36f2170b1787e416c0d6b42e689feb9553333ead7878b6c08ed213815a553f)
BibTeX
@misc{dce68411fd7f3f,
  title = {Kernel-Based Steering of CLIP with Vision-Language Model Preferences — arXiv:2609.32203},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32203},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e68411fd7f3f; sha256 dd36f2170b1787e416c0d6b42e689feb9553333ead7878b6c08ed213815a553f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32203. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large vision-language models (VLMs) can judge visual similarity, but their judgments are not directly available as compact image embeddings for efficient comparison. We study how to transfer these preferences into CLIP while retaining its image--text capabilities. We introduce ASK, a kernel-based steering method that learns from elicited pairwise judgments without accessing teacher embeddings or collecting new human similarity annotations. ASK constructs positive semidefinite target kernels within small image groups and combines visual kernel matching with an image--text distributional anchor. Low-rank adapters jointly update the visual and text encoders while regularizing predictions toward frozen CLIP. After adaptation, retrieval uses CLIP image embeddings and cosine similarity, with no VLM calls. Experiments across five image domains, four CLIP backbones, and six judges evaluate teacher agreement, retrieval, and recognition retention. For ViT-B/16, mean retrieval mAP on classes excluded from adaptation increases from 53.8 to 75.0, compared with 71.7 for DINOv2 targets with KL anchoring. Mean zero-shot accuracy with jointly adapted encoders increases from 61.8\% to 62.4\%, averaged over 12 benchmarks and the five adaptation domains. Prompting provides an additional capability: selecting which visual distinctions the student learns. Human-annotated evaluations across four datasets support this criterion-specific control.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32203 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:28.586Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).