Metraipolis › archive › dc_70b7fa64d651
SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features — arXiv:2608.10709
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.10709
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T06:19:25.808Z
- SHA-256 of the archived text
7c4a9a546cbc06ecd095fed6f66c709727ced51a2ed66f05769cdd1613efdbe3- Stable URL of this page
https://metraipolis.com/documento/dc_70b7fa64d651
How to cite
SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features — arXiv:2608.10709 — arXiv (en), https://arxiv.org/abs/2608.10709 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_70b7fa64d651 (Metraipolis archive dc_70b7fa64d651, sha256 7c4a9a546cbc06ecd095fed6f66c709727ced51a2ed66f05769cdd1613efdbe3)
BibTeX
@misc{dc70b7fa64d651,
title = {SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features — arXiv:2608.10709},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.10709},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_70b7fa64d651; sha256 7c4a9a546cbc06ecd095fed6f66c709727ced51a2ed66f05769cdd1613efdbe3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10709. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Quantization-Aware Training (QAT) enables the deployment of quantized models with minimal accuracy degradation. However, in practical scenarios, training labels are often unavailable due to privacy, copyright, or cost constraints. Knowledge Distillation (KD) is a common approach to address this challenge, but we observe that prior work combining QAT with KD suffers from a fundamental limitation: during distillation, the range mismatch between the teacher and the quantized student model induces an unattainable residual, resulting in an irreducible lower bound on the distillation loss. Motivated by this observation, we propose SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound by applying the student's quantization parameters to quantize the teacher's features during distillation. Through comprehensive experiments across diverse settings, we demonstrate that SQuaT consistently outperforms strong baselines, with particularly pronounced gains in extreme low-bit (e.g., 1- and 2-bit) settings. Furthermore, extensive evaluations across various model design choices show that our approach does not rely on specific architectural assumptions, making it broadly applicable across diverse architectures and quantization settings. The source code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.10709 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T06:11:45.092Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).