Metraipolis › archive › dc_94fc080260dc

Scaling Properties of Same-Family On-Policy Distillation — arXiv:2609.32722

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.32722
License (as archived): CC0-1.0
Archived: 2026-10-07T22:36:47.207Z
SHA-256 of the archived text: 2d064ae8095448d58ca223608445be739ce1be95adc5a9b6e034cdd642655642
Stable URL of this page: https://metraipolis.com/documento/dc_94fc080260dc

How to cite

Scaling Properties of Same-Family On-Policy Distillation — arXiv:2609.32722 — arXiv (en), https://arxiv.org/abs/2609.32722 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_94fc080260dc (Metraipolis archive dc_94fc080260dc, sha256 2d064ae8095448d58ca223608445be739ce1be95adc5a9b6e034cdd642655642)

BibTeX

@misc{dc94fc080260dc,
  title = {Scaling Properties of Same-Family On-Policy Distillation — arXiv:2609.32722},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32722},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_94fc080260dc; sha256 2d064ae8095448d58ca223608445be739ce1be95adc5a9b6e034cdd642655642},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32722. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

*Reinforcement learning (RL)* can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of *on-policy distillation (OPD)* across *weak-to-strong*, *same-base*, and *strong-to-weak* teacher--student setups. We find that early OPD training dynamics uniformly exhibit a regular *useful-transfer* regime, in which held-out accuracy (the *gold score*, $G$) rises approximately linearly in $d=\sqrt{\mathrm{KL}(\pi_\theta \Vert \pi_{\mathrm{ref}})}$, the square root of token-level reverse KL divergence from the student initialization. In every observed weak-to-strong pair, the student's peak gold score exceeds its teacher's own, so a compact RL expert can transfer capability to a much larger student via OPD. To estimate OPD outcomes, we fit *power laws* for how $G_{\mathrm{peak}}$ and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws show that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score smaller teachers transfer better, so a teacher's score alone does not define its supervision value. We also study the scaling effects of two OPD variants, bootstrapping weak-to-strong OPD, and the degree of on-policy supervision.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32722 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.747Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).