Metraipolis › archive › dc_a20ece9febcd

Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live? — arXiv:2610.19145

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.19145
License (as archived): CC0-1.0
Archived: 2026-10-15T14:15:12.815Z
SHA-256 of the archived text: c3a0848477128cbe0713395b2c5f9f6cbad7c44cec190c1af458d5a9a2ed1220
Stable URL of this page: https://metraipolis.com/documento/dc_a20ece9febcd

How to cite

Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live? — arXiv:2610.19145 — arXiv (en), https://arxiv.org/abs/2610.19145 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a20ece9febcd (Metraipolis archive dc_a20ece9febcd, sha256 c3a0848477128cbe0713395b2c5f9f6cbad7c44cec190c1af458d5a9a2ed1220)

BibTeX

@misc{dca20ece9febcd,
  title = {Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live? — arXiv:2610.19145},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19145},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a20ece9febcd; sha256 c3a0848477128cbe0713395b2c5f9f6cbad7c44cec190c1af458d5a9a2ed1220},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19145. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.19145 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.538Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).