Metraipolis › archive › dc_69c7fc24d4be
You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces — arXiv:2610.37937
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.37937
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-01T02:26:50.026Z
- SHA-256 of the archived text
7ef2ac8bd40aa2d387e6c3e0f7677f77694db79da3a71cdf78eac96edd202ffb- Stable URL of this page
https://metraipolis.com/documento/dc_69c7fc24d4be
How to cite
You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces — arXiv:2610.37937 — arXiv (en), https://arxiv.org/abs/2610.37937 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_69c7fc24d4be (Metraipolis archive dc_69c7fc24d4be, sha256 7ef2ac8bd40aa2d387e6c3e0f7677f77694db79da3a71cdf78eac96edd202ffb)
BibTeX
@misc{dc69c7fc24d4be,
title = {You Don't Need Aligned Representations: Knowledge Distillation via Random Prototype Spaces — arXiv:2610.37937},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.37937},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_69c7fc24d4be; sha256 7ef2ac8bd40aa2d387e6c3e0f7677f77694db79da3a71cdf78eac96edd202ffb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.37937. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A widely held assumption in knowledge distillation is that teacher and student representations must occupy a compatible or explicitly aligned feature space. We propose RPKD (Random Prototype Knowledge Distillation), which discards this assumption by comparing both networks against a shared bank of randomly initialized, frozen prototype vectors. Alignment is therefore carried out in an external reference space rather than in the teacher's native feature coordinates: the teacher side is projected by fixed random maps that carry no trainable parameters, and only the student adapts. Two branches handle the transfer: a logit branch aligning global prototype similarity distributions and a feature branch matching spatially-resolved prototype activation maps. Both operate within the same fixed prototype space, which can be interpreted as a random feature embedding that approximately preserves inter-sample similarity structure, so the prototype bank and the distillation objective stay unchanged across teacher-student pairs while only the input projections are instantiated per architecture. In low-category settings, decoupling the prototype vocabulary from the task label space yields richer supervision than the label space alone, an advantage absent from logit-based methods, whose supervisory signal collapses as class count shrinks. This advantage is especially relevant in real-world scenarios where the label space is typically constrained. Extensive experiments demonstrate the effectiveness of RPKD, achieving a maximum gain of +7.98% over OFA on CIFAR-100, +12.44% on ImageNet-100, and +1.28% on ImageNet-1K. Source code is available at this https URL Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.37937 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-01T02:12:44.236Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).