Metraipolis › archive › dc_53fe6aec3f33

When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation — arXiv:2608.27960

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27960
License (as archived)
CC0-1.0
Archived
2026-10-17T10:47:21.362Z
SHA-256 of the archived text
82b023215f7184dc28b770a5ef6cdb0d260deefb214d48d688217d56225f807a
Stable URL of this page
https://metraipolis.com/documento/dc_53fe6aec3f33

How to cite

When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation — arXiv:2608.27960 — arXiv (en), https://arxiv.org/abs/2608.27960 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_53fe6aec3f33 (Metraipolis archive dc_53fe6aec3f33, sha256 82b023215f7184dc28b770a5ef6cdb0d260deefb214d48d688217d56225f807a)
BibTeX
@misc{dc53fe6aec3f33,
  title = {When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation — arXiv:2608.27960},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27960},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_53fe6aec3f33; sha256 82b023215f7184dc28b770a5ef6cdb0d260deefb214d48d688217d56225f807a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27960. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) has recently emerged as a popular post-training paradigm for large language models (LLMs), providing an efficient way to transfer the knowledge and capabilities of teacher models into student models. However, teacher guidance on student-generated prefixes is not always reliable. Training should optimize the model to generate responses that are more likely to be correct, or equivalently, to get higher outcome rewards. But during OPD, the teacher model may provide guidance that discourages the student from moving toward correct trajectories or moves the student toward incorrect ones, which is misaligned with outcome reward. Such misaligned guidance is unreliable, as it would mislead the optimization process and ultimately degrade model performance. To mitigate misaligned teacher guidance, we propose Reward-Aligned On-Policy Distillation (RA-OPD). The key insight is to keep only trajectories whose induced updates move the student toward correct trajectories or discourage the student from moving toward incorrect ones. Specifically, for each sampled trajectory, RA-OPD checks whether its trajectory-level distillation return is consistent with its outcome reward and then filters out the misaligned trajectories. RA-OPD selects more reliable trajectories to improve student model performance without requiring additional computational cost. We evaluate RA-OPD on math and code benchmarks using models from the Qwen3 family and the DeepSeek-R1 family. Across seven math benchmarks and three code benchmarks, RA-OPD significantly outperforms standard OPD and other tested OPD variants.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.27960 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:33:42.374Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).