Metraipolis › archive › dc_2f5d428c718c

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives — arXiv:2609.38666

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38666
License (as archived)
CC0-1.0
Archived
2026-10-07T12:44:04.935Z
SHA-256 of the archived text
a539c82bcf83563d231466194a501abbd57f919e484d2866e91c09797085d5b2
Stable URL of this page
https://metraipolis.com/documento/dc_2f5d428c718c

How to cite

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives — arXiv:2609.38666 — arXiv (en), https://arxiv.org/abs/2609.38666 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2f5d428c718c (Metraipolis archive dc_2f5d428c718c, sha256 a539c82bcf83563d231466194a501abbd57f919e484d2866e91c09797085d5b2)
BibTeX
@misc{dc2f5d428c718c,
  title = {Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives — arXiv:2609.38666},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38666},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2f5d428c718c; sha256 a539c82bcf83563d231466194a501abbd57f919e484d2866e91c09797085d5b2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38666. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). However, its benefits and fragility remain incompletely understood. We study sequential distillation from multiple teachers, where the student minimizes its average divergence from the teachers. Forward Kullback--Leibler (KL) divergence yields a weighted arithmetic mixture, while reverse KL yields a normalized weighted geometric aggregate. We develop algorithms that learn these targets under off-policy and on-policy feedback, respectively, establishing logarithmic regret bounds in the tabular setting and extending the analysis to function approximation. By analyzing these aggregation targets, we identify mechanisms that help explain both the benefits and fragility of OPD. Relative to forward KL, reverse KL can better retain a confident expert's preferences under uninformative feedback, but is more sensitive to teachers that assign very low probabilities to correct responses. Its token-level conditionals also reveal a dependence on continuation distributions that can favor incorrect prefixes over long horizons.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.38666 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.918Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).