Metraipolis › archive › dc_8d268563eb0c
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models — arXiv:2609.38025
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38025
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:28:01.979Z
- SHA-256 of the archived text
5bb358d725592da14a0062252367e0e245c0d931a0b8053c1354d4f22ae7d559- Stable URL of this page
https://metraipolis.com/documento/dc_8d268563eb0c
How to cite
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models — arXiv:2609.38025 — arXiv (en), https://arxiv.org/abs/2609.38025 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d268563eb0c (Metraipolis archive dc_8d268563eb0c, sha256 5bb358d725592da14a0062252367e0e245c0d931a0b8053c1354d4f22ae7d559)
BibTeX
@misc{dc8d268563eb0c,
title = {Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models — arXiv:2609.38025},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38025},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d268563eb0c; sha256 5bb358d725592da14a0062252367e0e245c0d931a0b8053c1354d4f22ae7d559},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38025. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by $9.7$ points over vanilla OPD, and enables the smaller student to surpass its larger teacher. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Optimization and Control (math.OC). Pagina del lavoro: https://arxiv.org/abs/2609.38025 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T15:54:54.062Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).