Metraipolis › archive › dc_2eb7f723d2ca
The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation — arXiv:2609.34106
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34106
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:02:07.717Z
- SHA-256 of the archived text
775374535b4636155f1303b75f1261b7815b7a7da5f119a6782692c2639ba2e7- Stable URL of this page
https://metraipolis.com/documento/dc_2eb7f723d2ca
How to cite
The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation — arXiv:2609.34106 — arXiv (en), https://arxiv.org/abs/2609.34106 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2eb7f723d2ca (Metraipolis archive dc_2eb7f723d2ca, sha256 775374535b4636155f1303b75f1261b7815b7a7da5f119a6782692c2639ba2e7)
BibTeX
@misc{dc2eb7f723d2ca,
title = {The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation — arXiv:2609.34106},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34106},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2eb7f723d2ca; sha256 775374535b4636155f1303b75f1261b7815b7a7da5f119a6782692c2639ba2e7},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34106. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large visual foundation models have demonstrated remarkable transferability across a wide range of downstream tasks. To deploy such models efficiently, feature matching has become a popular knowledge distillation approach that transfers teacher representations to smaller student models without requiring labeled data. However, we show that the conventional feature matching objective with L2-distance is inherently biased toward reconstructing dominant spectral directions of the teacher representation, while under-optimizing low-variance directions that often contain task-relevant information. To address this, we propose Spectrum-Balanced Feature Matching, SpecMatch, a simple objective that adaptively emphasizes under-optimized spectral directions while preserving the relative importance of dominant directions. SpecMatch is easy to implement and introduces negligible computational overhead. Extensive experiments on image recognition demonstrate that SpecMatch consistently improves downstream adaptation across diverse tasks, including image classification, anomaly detection, medical image analysis, and domain generalization. In particular, SpecMatch outperforms conventional feature matching in 40 of 42 teacher--student and training-setting combinations, while consistently improving over the original student model in all settings. We further demonstrate that the proposed objective generalizes beyond vision, improving downstream performance across six protein understanding tasks. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.34106 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:02.855Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).