Metraipolis › archive › dc_67c8b352ba29

Diagnosing On-Policy Self-Distillation for Reasoning Language Models — arXiv:2609.39118

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39118
License (as archived)
CC0-1.0
Archived
2026-10-07T13:13:42.160Z
SHA-256 of the archived text
e954ae5730817971a03742d603bb15f64531eaa1a8627b93f3f49fdf2a419382
Stable URL of this page
https://metraipolis.com/documento/dc_67c8b352ba29

How to cite

Diagnosing On-Policy Self-Distillation for Reasoning Language Models — arXiv:2609.39118 — arXiv (en), https://arxiv.org/abs/2609.39118 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_67c8b352ba29 (Metraipolis archive dc_67c8b352ba29, sha256 e954ae5730817971a03742d603bb15f64531eaa1a8627b93f3f49fdf2a419382)
BibTeX
@misc{dc67c8b352ba29,
  title = {Diagnosing On-Policy Self-Distillation for Reasoning Language Models — arXiv:2609.39118},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39118},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_67c8b352ba29; sha256 e954ae5730817971a03742d603bb15f64531eaa1a8627b93f3f49fdf2a419382},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39118. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B--8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.39118 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:44:51.798Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).