Metraipolis › archive › dc_7db0da340eb7

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models — arXiv:2609.36734

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.36734
License (as archived): CC0-1.0
Archived: 2026-10-07T16:56:41.867Z
SHA-256 of the archived text: f62ad4350d6444ef86180cfb397002d35d87d592a6dcedaf79dcdf236a78696b
Stable URL of this page: https://metraipolis.com/documento/dc_7db0da340eb7

How to cite

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models — arXiv:2609.36734 — arXiv (en), https://arxiv.org/abs/2609.36734 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7db0da340eb7 (Metraipolis archive dc_7db0da340eb7, sha256 f62ad4350d6444ef86180cfb397002d35d87d592a6dcedaf79dcdf236a78696b)

BibTeX

@misc{dc7db0da340eb7,
  title = {Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models — arXiv:2609.36734},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36734},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7db0da340eb7; sha256 f62ad4350d6444ef86180cfb397002d35d87d592a6dcedaf79dcdf236a78696b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36734. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implicitly assuming the teacher to be a reliable oracle. In large language models (LLMs), this assumption often fails: teacher predictions can exhibit high entropy and hallucinations, causing standard KD to degrade well-calibrated student priors. We propose CaRE-KD, a confidence-gated distillation framework that replaces static objectives with uncertainty-adaptive optimization. CaRE-KD has two components: a token-level loss (CaRE-Divergence) that adaptively switches between Forward and Reverse KL divergence based on teacher--student confidence, and a batch-level epistemic rejection mechanism (Revival) that suppresses updates when the teacher is more uncertain than the student. We provide a gradient-level analysis showing how this dual-granularity design induces a conditional calibration mechanism that prior static divergences cannot reproduce. Empirically, across eight teacher--student pairs and eleven benchmarks spanning instruction following, chat alignment, code generation, and mathematical reasoning, CaRE-KD delivers consistent gains over strong baselines (Skewed-KL, $\alpha$--$\beta$ divergence). Highlights include up to $+3.2$ average ROUGE-L on instruction-following tasks, $+2.1$ pass@1 on MBPP, $+1.7$ accuracy on GSM8k, and $+1.8$ accuracy on CollegeMath over the strongest baseline, with consistent gains in LLM-as-a-judge factuality (up to $+2.5$ per task over Skewed-RKL). Revival further acts as a principled, loss-agnostic plug-in that systematically strengthens existing distillation objectives by filtering epistemically unreliable teacher supervision.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.36734 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:27:03.240Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).