Metraipolis › archive › dc_aa208963c5dd
TV-Regulated OPD: Direction Matters in On-Policy Distillation — arXiv:2609.08341
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.08341
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:58:13.724Z
- SHA-256 of the archived text
8fbefa0dfc25e5774e6ab6d0d383b99634cc2e37eff4fc21308335015e5b7c59- Stable URL of this page
https://metraipolis.com/documento/dc_aa208963c5dd
How to cite
TV-Regulated OPD: Direction Matters in On-Policy Distillation — arXiv:2609.08341 — arXiv (en), https://arxiv.org/abs/2609.08341 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aa208963c5dd (Metraipolis archive dc_aa208963c5dd, sha256 8fbefa0dfc25e5774e6ab6d0d383b99634cc2e37eff4fc21308335015e5b7c59)
BibTeX
@misc{dcaa208963c5dd,
title = {TV-Regulated OPD: Direction Matters in On-Policy Distillation — arXiv:2609.08341},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.08341},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aa208963c5dd; sha256 8fbefa0dfc25e5774e6ab6d0d383b99634cc2e37eff4fc21308335015e5b7c59},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08341. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-Policy Distillation (OPD) facilitates the transfer of knowledge from domain expert to student in the post-training phase of Large Language Models (LLMs). However, the supervision signals in mainstream OPD methods suffer from high variance and noise which is generally instable during training. In this work, we systematically investigated what really matters to the performance and the fundamental mechanisms behind the instability during training. We found that retaining only the sign of token-level advantages is sufficient to achieve the performance comparable to standard OPD. Meanwhile, smoother and bounded advantages can stabilize the training process without sacrificing its performance. These motivated us to shape the advantages using the Total Variation (TV) and propose a robust TV regulated On-Policy Distillation (TV-OPD) method. Benefiting from the bounded and diminished advantages, TV-OPD exhibits stable training dynamics and steady late-stage performance. We conducted comprehensive experiments and found that, across various settings, TV-OPD consistently achieved better performance and lower variance in the late-stage of training. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.08341 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:39:43.036Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).