Metraipolis › archive › dc_b4d8ea87e497

Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention — arXiv:2609.39494

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39494
License (as archived)
CC0-1.0
Archived
2026-10-07T12:42:49.154Z
SHA-256 of the archived text
aea47769281ddb6affe8050be9815e57f31699f9e46f3e94fe251a9cf16b296d
Stable URL of this page
https://metraipolis.com/documento/dc_b4d8ea87e497

How to cite

Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention — arXiv:2609.39494 — arXiv (en), https://arxiv.org/abs/2609.39494 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b4d8ea87e497 (Metraipolis archive dc_b4d8ea87e497, sha256 aea47769281ddb6affe8050be9815e57f31699f9e46f3e94fe251a9cf16b296d)
BibTeX
@misc{dcb4d8ea87e497,
  title = {Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention — arXiv:2609.39494},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39494},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b4d8ea87e497; sha256 aea47769281ddb6affe8050be9815e57f31699f9e46f3e94fe251a9cf16b296d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39494. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Self-distillation with privileged context adapts a language model from demonstrations by letting the model, once conditioned on a reference response, teach its context-free copy token by token. Our taxonomy reveals existing methods differ along three entangled axes: (i) the rollout source (student or teacher), (ii) the teacher coupling (frozen, or an exponential moving average of the student at some coupling rate) and (iii) the KL direction (reverse or forward), yet these axes are usually studied in fixed combinations and have led to conflicting conclusions. We formalize a unifying framework to encompass all self-distillation methods vs classic supervised fine-tuning: we train every combination of the three axes, on Qwen2.5-7B and Ministral-3-3B across ordinary and contradictory tasks, totaling 1,200 adaptation runs, to systematically investigate the impact of the above axes. We propose a controlled model of the same objective to explain the resulting acquisition-retention trade-offs. We find that (i) the rollout source matters mostly where the task contradicts the pretrained behavior: there teacher rollouts raise acquisition well above what student rollouts achieve, with almost no change in retention; (ii) the teacher coupling changes acquisition most, on every task: acquisition rises with the coupling rate, then falls past a task-specific rate; (iii) switching the KL direction costs retention in one model but not the other so which axis to tune first depends on the model. The controlled model reproduces the three trends.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.39494 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.768Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).