Metraipolis › archive › dc_bcf08616cea8

When to Follow the Teacher: Advantage-Based On-Policy Distillation — arXiv:2610.31563

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.31563
License (as archived)
CC0-1.0
Archived
2026-10-23T15:16:44.350Z
SHA-256 of the archived text
b5d9ddb17d0fd08410b9e95e2f9b6fc8e9c1e01ab591f55b4056eb8ed0985f69
Stable URL of this page
https://metraipolis.com/documento/dc_bcf08616cea8

How to cite

When to Follow the Teacher: Advantage-Based On-Policy Distillation — arXiv:2610.31563 — arXiv (en), https://arxiv.org/abs/2610.31563 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bcf08616cea8 (Metraipolis archive dc_bcf08616cea8, sha256 b5d9ddb17d0fd08410b9e95e2f9b6fc8e9c1e01ab591f55b4056eb8ed0985f69)
BibTeX
@misc{dcbcf08616cea8,
  title = {When to Follow the Teacher: Advantage-Based On-Policy Distillation — arXiv:2610.31563},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31563},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bcf08616cea8; sha256 b5d9ddb17d0fd08410b9e95e2f9b6fc8e9c1e01ab591f55b4056eb8ed0985f69},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31563. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Online Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every state of the trajectory, assuming that teacher guidance is always beneficial. We instead distinguish between what the teacher prefers and what actually helps the current student succeed. We introduce Advantage-Based On-Policy Distillation (AOPD), which weights teacher actions by their counterfactual advantage under the student. Teacher actions that improve the student's expected outcome are reinforced, while those that do not are downweighted. Directly computing these advantages requires trying alternative teacher actions at each step and running new rollouts from each. Instead, we show that the AOPD objective can be estimated using only standard student rollouts and their terminal rewards. Our estimator uses teacher importance weights on the student-sampled actions to provide unbiased advantage-weighted supervision, without any additional rollouts or reward evaluations. This allows AOPD to focus teacher supervision on actions that help the current student while preserving the rollout efficiency of standard OPD. Experiments on math, coding, and Medqa show gains over distillation and RL baselines in both external-teacher and self-distillation settings.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.31563 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-23T15:09:08.632Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).