Metraipolis › archive › dc_fbac0d35b540
Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models — arXiv:2609.37132
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.37132
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:23:31.840Z
- SHA-256 of the archived text
0f1c0b9009fc31220dc1dd56bc3da1e20407e23f3c6ef6dea90ae55fa20d3310- Stable URL of this page
https://metraipolis.com/documento/dc_fbac0d35b540
How to cite
Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models — arXiv:2609.37132 — arXiv (en), https://arxiv.org/abs/2609.37132 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbac0d35b540 (Metraipolis archive dc_fbac0d35b540, sha256 0f1c0b9009fc31220dc1dd56bc3da1e20407e23f3c6ef6dea90ae55fa20d3310)
BibTeX
@misc{dcfbac0d35b540,
title = {Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models — arXiv:2609.37132},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.37132},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbac0d35b540; sha256 0f1c0b9009fc31220dc1dd56bc3da1e20407e23f3c6ef6dea90ae55fa20d3310},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37132. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level supervision on the model's own trajectories. Yet existing methods typically construct the self-teacher from the current, initial, or slowly averaged policy state, leaving the quality of supervision constrained by the teacher's ability to exploit privileged information. We ask whether the model's own optimization progress can instead be recycled into a stronger self-teacher. In this paper, we introduce Bootstrapped On-Policy Self-Distillation (B-OPSD), which temporarily trains the policy ahead to obtain a future teacher, restores the student to the original policy state, and then uses the future teacher to supervise the restarted student. The future teacher improves supervision in two complementary ways, it can generate more reliable privileged trajectories and, conditioned on them, provide more informative token-level targets along the restarted student's on-policy trajectories. Experiments on mathematical reasoning with Qwen3-4B and Qwen3-8B show consistent improvements over standard OPSD in both settings, including gains from 27.50 to 41.30 and from 48.80 to 64.44 in the rollout-privileged setting. Our findings point to a broader principle for self-improving models that future learning progress can be distilled backward, preserving acquired knowledge while bootstrapping beyond the optimization state that produced it. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.37132 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T15:54:53.592Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).