Metraipolis › archive › dc_4643dc4ae05d
RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods — arXiv:2611.04081
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04081
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T18:53:58.680Z
- SHA-256 of the archived text
ff614ce089e8fe7a6a89af26939d3a75eccbcf4664c2a2fad993d347afeb7a7c- Stable URL of this page
https://metraipolis.com/documento/dc_4643dc4ae05d
How to cite
RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods — arXiv:2611.04081 — arXiv (en), https://arxiv.org/abs/2611.04081 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4643dc4ae05d (Metraipolis archive dc_4643dc4ae05d, sha256 ff614ce089e8fe7a6a89af26939d3a75eccbcf4664c2a2fad993d347afeb7a7c)
BibTeX
@misc{dc4643dc4ae05d,
title = {RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods — arXiv:2611.04081},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04081},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4643dc4ae05d; sha256 ff614ce089e8fe7a6a89af26939d3a75eccbcf4664c2a2fad993d347afeb7a7c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04081. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Knowledge distillation is a technique used to transfer the expertise of large teacher models into small, efficient student models. Recent advances have integrated the distillation of internal activations, more directly training students to learn similar representations as the teacher. One of the core challenges of internal distillation is the dimensional incompatibility between the student and teacher internal representations. Existing methods add learnable matrices to project the larger teacher activations into the smaller representation space of the student. This is a notable point of inefficiency, as learning additional parameters for the projectors increases training costs, only for them to be discarded immediately after training. We introduce RAPID (RAndomized Projection for Internal Distillation) a novel distillation methodology which eliminates the wasteful projection learning by leveraging randomized methods. RAPID utilizes an efficient transform construction based on the Johnson-Lindenstrauss lemma to compress the teacher representations into the student embedding space with strong theoretical guarantees of structural preservation. RAPID also employs a combination of exponential moving average (EMA) and homoscedastic uncertainty weighting in a novel multi-objective function designed for internal distillation. Our experiments in post-training for reasoning demonstrate that RAPID increases training efficiency by as much as 21.1% while maintaining performance comparable to existing methods. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.04081 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T18:35:37.423Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).