Metraipolis › archive › dc_4aede0299946

Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models — arXiv:2608.17306

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.17306
License (as archived)
CC0-1.0
Archived
2026-10-18T14:31:26.409Z
SHA-256 of the archived text
40fb9d76dee2294a896be06e038c8d6ad3c4ca842434500a69f78f1a10f7c148
Stable URL of this page
https://metraipolis.com/documento/dc_4aede0299946

How to cite

Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models — arXiv:2608.17306 — arXiv (en), https://arxiv.org/abs/2608.17306 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4aede0299946 (Metraipolis archive dc_4aede0299946, sha256 40fb9d76dee2294a896be06e038c8d6ad3c4ca842434500a69f78f1a10f7c148)
BibTeX
@misc{dc4aede0299946,
  title = {Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models — arXiv:2608.17306},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.17306},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4aede0299946; sha256 40fb9d76dee2294a896be06e038c8d6ad3c4ca842434500a69f78f1a10f7c148},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.17306. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-19 in cs.AI).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.17306 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T14:27:54.414Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-19?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).