Metraipolis › archive › dc_391cd49cd574

Reward-Aligned Reweighting for On-Policy Distillation — arXiv:2609.35517

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.35517
License (as archived): CC0-1.0
Archived: 2026-10-07T22:39:11.897Z
SHA-256 of the archived text: b5b6290773f64fc8ad698aaccd836ccceaacc1a46ffd4a38fffb4381af925f18
Stable URL of this page: https://metraipolis.com/documento/dc_391cd49cd574

How to cite

Reward-Aligned Reweighting for On-Policy Distillation — arXiv:2609.35517 — arXiv (en), https://arxiv.org/abs/2609.35517 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_391cd49cd574 (Metraipolis archive dc_391cd49cd574, sha256 b5b6290773f64fc8ad698aaccd836ccceaacc1a46ffd4a38fffb4381af925f18)

BibTeX

@misc{dc391cd49cd574,
  title = {Reward-Aligned Reweighting for On-Policy Distillation — arXiv:2609.35517},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35517},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_391cd49cd574; sha256 b5b6290773f64fc8ad698aaccd836ccceaacc1a46ffd4a38fffb4381af925f18},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35517. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) trains a student language model with dense feedback from a stronger teacher on student-generated trajectories. Yet standard OPD weights token-level distillation terms uniformly, implicitly treating local teacher preference as a proxy for correction utility. A decision's task value, however, depends on how the student completes the subsequent reasoning. This mismatch can cause imitation to suppress viable student strategies or reinforce paths the student cannot reliably execute. Verified trajectory outcomes provide complementary evidence about continuation quality, but do not directly identify the utility of individual decisions. We introduce Reward-Aligned Reweighting for On-Policy Distillation (R$^{2}$-OPD), which uses outcome agreement and the magnitude of teacher--student disagreement to continuously reallocate teacher supervision. It gives reward-aligned corrections greater relative influence while retaining dense feedback, moving beyond uniform imitation and hard filtering. Our analysis formalizes the mismatch between local teacher preference and student continuation value and establishes sufficient conditions for reallocation to improve first-order task progress over uniform OPD. Across seven mathematical reasoning benchmarks, R$^{2}$-OPD achieves the highest average accuracy among the compared training methods in both cross-size and same-size distillation. It outperforms standard OPD on all seven benchmarks, with average gains of 3.5 and 2.4 percentage points for 1.7B and 4B students, respectively. An extension to code generation yields an average gain of 1.6 percentage points over standard OPD. These results highlight outcome-guided supervision allocation as an effective way to translate dense teacher feedback into stronger student performance across model scales and task domains.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.35517 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:28.464Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).