Metraipolis › archive › dc_691317ccc992
The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation — arXiv:2609.36484
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36484
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:29:05.829Z
- SHA-256 of the archived text
e3692012fa5b668a5b4c609617575605e97d792f28723fcdbaa8df1cd5664011- Stable URL of this page
https://metraipolis.com/documento/dc_691317ccc992
How to cite
The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation — arXiv:2609.36484 — arXiv (en), https://arxiv.org/abs/2609.36484 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_691317ccc992 (Metraipolis archive dc_691317ccc992, sha256 e3692012fa5b668a5b4c609617575605e97d792f28723fcdbaa8df1cd5664011)
BibTeX
@misc{dc691317ccc992,
title = {The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation — arXiv:2609.36484},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36484},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_691317ccc992; sha256 e3692012fa5b668a5b4c609617575605e97d792f28723fcdbaa8df1cd5664011},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36484. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The language-model head, however, attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at a small fraction of its weight, and the sampled-token log-probability ratios on which output-space extrapolation relies inject noise that the extrapolation amplifies, making training unstable. We observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Motivated by this observation, we propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space: at every layer and token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual. Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher, which makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher. Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so, and it consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. Project page: this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.36484 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.171Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).