Metraipolis › archive › dc_ef26ed68f326

RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods — arXiv:2611.04081

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04081
License (as archived)
CC0-1.0
Archived
2026-11-04T12:40:42.704Z
SHA-256 of the archived text
10bf22b68a2d7d5cbf31db2b0de5952768cacd41379fd8395d648c75f6e536d5
Stable URL of this page
https://metraipolis.com/documento/dc_ef26ed68f326

How to cite

RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods — arXiv:2611.04081 — arXiv (en), https://arxiv.org/abs/2611.04081 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef26ed68f326 (Metraipolis archive dc_ef26ed68f326, sha256 10bf22b68a2d7d5cbf31db2b0de5952768cacd41379fd8395d648c75f6e536d5)
BibTeX
@misc{dcef26ed68f326,
  title = {RAPID: Increasing Efficiency of Internal Distillation via Randomized Methods — arXiv:2611.04081},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04081},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef26ed68f326; sha256 10bf22b68a2d7d5cbf31db2b0de5952768cacd41379fd8395d648c75f6e536d5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04081. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Knowledge distillation is a technique used to transfer the expertise of large teacher models into small, efficient student models. Recent advances have integrated the distillation of internal activations, more directly training students to learn similar representations as the teacher. One of the core challenges of internal distillation is the dimensional incompatibility between the student and teacher internal representations. Existing methods add learnable matrices to project the larger teacher activations into the smaller representation space of the student. This is a notable point of inefficiency, as learning additional parameters for the projectors increases training costs, only for them to be discarded immediately after training. We introduce RAPID (RAndomized Projection for Internal Distillation) a novel distillation methodology which eliminates the wasteful projection learning by leveraging randomized methods. RAPID utilizes an efficient transform construction based on the Johnson-Lindenstrauss lemma to compress the teacher representations into the student embedding space with strong theoretical guarantees of structural preservation. RAPID also employs a combination of exponential moving average (EMA) and homoscedastic uncertainty weighting in a novel multi-objective function designed for internal distillation. Our experiments in post-training for reasoning demonstrate that RAPID increases training efficiency by as much as 21.1% while maintaining performance comparable to existing methods.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04081 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:29:47.274Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).