Metraipolis › archive › dc_308f698fd698
WASD: Wasserstein-based Knowledge Distillation for Large Language Models — arXiv:2610.07706
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07706
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:21:05.744Z
- SHA-256 of the archived text
00bfb3adfcb388b1c062aea6d6517addbcf7d506f32394c937171d281aa88f4d- Stable URL of this page
https://metraipolis.com/documento/dc_308f698fd698
How to cite
WASD: Wasserstein-based Knowledge Distillation for Large Language Models — arXiv:2610.07706 — arXiv (en), https://arxiv.org/abs/2610.07706 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_308f698fd698 (Metraipolis archive dc_308f698fd698, sha256 00bfb3adfcb388b1c062aea6d6517addbcf7d506f32394c937171d281aa88f4d)
BibTeX
@misc{dc308f698fd698,
title = {WASD: Wasserstein-based Knowledge Distillation for Large Language Models — arXiv:2610.07706},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07706},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_308f698fd698; sha256 00bfb3adfcb388b1c062aea6d6517addbcf7d506f32394c937171d281aa88f4d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07706. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Autoregressive large language models (LLMs) have rapidly advanced in capability, but their increasing scale comes with substantial computational and memory costs at inference time. Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large teacher model to a smaller student model via alignment of discrete probability distributions. However, existing KD methods for LLMs primarily rely on divergences that evaluate discrepancies through probability values at each vocabulary index, without explicitly leveraging token-level semantic information. We propose Wasserstein-based knowledge distillation (WASD) for LLMs, which incorporates token-level semantic information via the Wasserstein-based distance with a cost matrix derived from token embeddings. To ensure computational tractability, we adopt the Sinkhorn divergence and derive a gradient-equivalent objective that can be efficiently optimized without introducing additional networks. Experiments across multiple LLM families and scales show that WASD consistently improves distillation performance on diverse tasks, including instruction following, mathematical reasoning, and code generation. Our results highlight the importance of semantic information encoded in the token space for effective distribution alignment in LLM distillation. The implementation is publicly available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.07706 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:24.756Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).