Metraipolis › archive › dc_a67abe831fef

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation — arXiv:2609.39338

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.39338
License (as archived): CC0-1.0
Archived: 2026-10-07T13:46:05.909Z
SHA-256 of the archived text: b40d03f0752853b833c9cc6e55816aaed4a56881c4a05e31c44b756090ac813b
Stable URL of this page: https://metraipolis.com/documento/dc_a67abe831fef

How to cite

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation — arXiv:2609.39338 — arXiv (en), https://arxiv.org/abs/2609.39338 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a67abe831fef (Metraipolis archive dc_a67abe831fef, sha256 b40d03f0752853b833c9cc6e55816aaed4a56881c4a05e31c44b756090ac813b)

BibTeX

@misc{dca67abe831fef,
  title = {Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation — arXiv:2609.39338},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39338},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a67abe831fef; sha256 b40d03f0752853b833c9cc6e55816aaed4a56881c4a05e31c44b756090ac813b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39338. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the correct class, but also relations among incorrect alternatives. Yet closer imitation does not necessarily yield a better student. A student may already distinguish the correct class more sharply than its teacher, so further imitation can require giving back discrimination it has acquired. Our main result is an exact separation between full imitation and conditional learning. When the correct class's score advantage over each alternative must be preserved, full teacher-to-student KL minimization is blocked exactly when the student assigns no more probability than the teacher to every incorrect class. Crucially, the teacher's relative probabilities among incorrect classes remain fully learnable. We characterize the exact price of this transfer: a minimum increase in correct-class log-odds that compensates for the largest conditional-probability mismatch. Label fitting and conditional matching can therefore be completed even as full teacher KL diverges. This separation motivates task-preserving knowledge distillation (TPKD), which keeps the label gradient intact and minimally corrects the conditional gradient so that its output update preserves the label step's gains against every incorrect alternative. The corrected conditional direction retains more than half of the original first-order conditional descent at the same step size, with a tight bound. For a fixed positive conditional target and sufficiently small constant output steps, label and conditional errors vanish together. Experiments trace this learning from exact head updates to ordinary network training. TPKD reaches 88.05% accuracy on CIFAR-100 and 93.81% on CLINC150, improving over standard distillation by 0.47 and 0.35 percentage points across three seeds.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.39338 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T13:18:10.366Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).