Metraipolis › archive › dc_aa48e01443a8
Diagnosing Zero-Shot Forgetting in CLIP Fine-Tuning: Attention Drift Characterizes Adaptation, Representation Drift Predicts Transfer Degradation — arXiv:2611.00042
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00042
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:39:15.522Z
- SHA-256 of the archived text
b81b6d3a213e886fabff7a18f262aa6a944746766b7d3dd5829255428f4acb04- Stable URL of this page
https://metraipolis.com/documento/dc_aa48e01443a8
How to cite
Diagnosing Zero-Shot Forgetting in CLIP Fine-Tuning: Attention Drift Characterizes Adaptation, Representation Drift Predicts Transfer Degradation — arXiv:2611.00042 — arXiv (en), https://arxiv.org/abs/2611.00042 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aa48e01443a8 (Metraipolis archive dc_aa48e01443a8, sha256 b81b6d3a213e886fabff7a18f262aa6a944746766b7d3dd5829255428f4acb04)
BibTeX
@misc{dcaa48e01443a8,
title = {Diagnosing Zero-Shot Forgetting in CLIP Fine-Tuning: Attention Drift Characterizes Adaptation, Representation Drift Predicts Transfer Degradation — arXiv:2611.00042},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00042},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aa48e01443a8; sha256 b81b6d3a213e886fabff7a18f262aa6a944746766b7d3dd5829255428f4acb04},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00042. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Fine-tuning CLIP on a downstream task typically reduces its zero-shot accuracy on other tasks, and quantifying this loss ordinarily requires a labeled transfer benchmark. We study whether the loss can be anticipated from unlabeled target images that are already available during fine-tuning. Four signals are compared: the entropy of CLS token attention, linear centered kernel alignment (CKA) with the pretrained model, embedding drift, and relative weight change. ViT-B/32 is adapted by full fine-tuning (Full FT) and by LoRA on EuroSAT and Oxford-IIIT Pets over four learning rates and three seeds, with an additional ViT-B/16 grid; DTD and GTSRB are held out as confirmation targets, and forgetting is measured as zero-shot retention on CIFAR-100, Food-101, and ImageNet-R. Within a fixed configuration, the change in attention entropy is strongly associated with forgetting (mean $|\rho|=0.860$). Its sign, however, depends on the dataset, the adaptation method, and the backbone, and the pooled correlation falls to 0.361. A block decomposition shows that this sign is determined almost entirely by blocks 7 to 12. CKA ranks runs by retention at $\rho=0.960$, and the ranking is already established after the first of six epochs ($\rho=0.959$). On the held-out targets, CKA reaches $\rho=0.95$, and the block account correctly predicts the sign of the attention change for both datasets. These results support a simple decision rule: CKA is suited to comparisons across configurations, whereas attention entropy is informative only within a single configuration. The evidence is limited to one model family and short training schedules. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.00042 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:12:54.668Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).