Metraipolis › archive › dc_fb01c299459a

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model — arXiv:2608.18339

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.18339
License (as archived): CC0-1.0
Archived: 2026-10-18T13:45:01.212Z
SHA-256 of the archived text: b0a89062f2005901fbd109943e28e0134e713b048eed180763572c2900ea8652
Stable URL of this page: https://metraipolis.com/documento/dc_fb01c299459a

How to cite

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model — arXiv:2608.18339 — arXiv (en), https://arxiv.org/abs/2608.18339 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb01c299459a (Metraipolis archive dc_fb01c299459a, sha256 b0a89062f2005901fbd109943e28e0134e713b048eed180763572c2900ea8652)

BibTeX

@misc{dcfb01c299459a,
  title = {From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model — arXiv:2608.18339},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18339},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb01c299459a; sha256 b0a89062f2005901fbd109943e28e0134e713b048eed180763572c2900ea8652},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18339. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference. Although significant efforts are devoted to adapting VLMs at test time, they rely heavily on noisy pseudo-labels predicted directly from raw embedding similarities during inference, which are unreliable under distribution shift and mislead the adaptation. To avoid noise amplification, existing works craft coarse-grained surrogate objectives during adaptation, which fail to explicitly model sample-level relationships across different modalities, creating objective mismatch with inference, thus leading to marginal performance improvement. In this work, we aim to bridge the detached objectives of inference and adaptation for VLMs, and propose a principled VLM TTA method called \algname. For VLM inference, we formulate the zero-shot image classification task as a cross-modal alignment problem encoded via a Wasserstein OT formulation, providing robust pseudo-labels at the sample-level to effectively adapt VLMs. For VLM adaptation, we adopt a soft-label InfoNCE loss to adapt VLMs based on the OT-induced pseudo-labels, leveraging fine-grained supervisions to explicitly model relationships of individual image-text pairs via contrastive learning, which empowers accurate inference at the same granularity. Moreover, we theoretically reveal that the InfoNCE loss can be neatly reformulated as a Wasserstein OT formulation, thereby unifying the objectives of the inference and adaptation of VLMs to achieve their mutual benefits. Extensive experiments demonstrate the effectiveness and efficiency of our methods, outperforming the best-performing methods by up to 7% with state-of-the-art efficiency.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.AI).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.18339 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:41:17.929Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).