Metraipolis › archive › dc_fa5901493225
Recursive Self-Improvement via On-Policy Distillation for Reasoning — arXiv:2609.30652
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.30652
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T02:03:55.924Z
- SHA-256 of the archived text
782e0a58c0c30de6844b3e7363adf95b31c1778b057f5f75a0346f93ecfb2386- Stable URL of this page
https://metraipolis.com/documento/dc_fa5901493225
How to cite
Recursive Self-Improvement via On-Policy Distillation for Reasoning — arXiv:2609.30652 — arXiv (en), https://arxiv.org/abs/2609.30652 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa5901493225 (Metraipolis archive dc_fa5901493225, sha256 782e0a58c0c30de6844b3e7363adf95b31c1778b057f5f75a0346f93ecfb2386)
BibTeX
@misc{dcfa5901493225,
title = {Recursive Self-Improvement via On-Policy Distillation for Reasoning — arXiv:2609.30652},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.30652},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa5901493225; sha256 782e0a58c0c30de6844b3e7363adf95b31c1778b057f5f75a0346f93ecfb2386},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30652. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher's next-token predictions. This provides dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE). Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model's own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL). Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.30652 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T01:38:49.432Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).