Metraipolis › archive › dc_d2fb3f863288

On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation — arXiv:2611.05609

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05609
License (as archived)
CC0-1.0
Archived
2026-11-05T18:21:58.105Z
SHA-256 of the archived text
2fb2d6f2f20ed3f2a14b41356b004fd97d2eebe35a534573ad8f1847955d03ec
Stable URL of this page
https://metraipolis.com/documento/dc_d2fb3f863288

How to cite

On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation — arXiv:2611.05609 — arXiv (en), https://arxiv.org/abs/2611.05609 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d2fb3f863288 (Metraipolis archive dc_d2fb3f863288, sha256 2fb2d6f2f20ed3f2a14b41356b004fd97d2eebe35a534573ad8f1847955d03ec)
BibTeX
@misc{dcd2fb3f863288,
  title = {On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation — arXiv:2611.05609},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05609},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d2fb3f863288; sha256 2fb2d6f2f20ed3f2a14b41356b004fd97d2eebe35a534573ad8f1847955d03ec},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05609. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) to consolidate capabilities from multiple teachers. Although prior work reports near-perfect in-distribution (IND) distillation performance, we find that this can mask incomplete capability consolidation beyond the supervised distribution. A preliminary study on synthetic reasoning shows that OPD does not fully transfer teachers' generalization capabilities. We then extend this investigation to real-world tasks, evaluating extrapolative and compositional generalization across mathematical reasoning, coding, agent, and long-context capabilities using Qwen3 models. Across extrapolative settings, OPD and MOPD reach 96.04-102.73% of their teachers' IND scores. Yet their out-of-distribution (OOD) scores remain 2.45-16.25% below the base model's. Across three compositional settings, MOPD improves IND performance over the base model, but its performance on composed tasks as OOD remains 18.19-24.34% below that of the best teachers. To understand these gaps, we examine teacher capability and data coverage as two potential bottlenecks. Jointly alleviating these bottlenecks eliminates OOD forgetting in our experiments and suggests the potential for generalization beyond the teacher. (M)OPD also reduces OOD forgetting on average relative to its reinforcement learning (RL) baselines. Analysis of the associated update geometry reveals smaller and more spectrally concentrated updates on average than those of the RL baselines, along with greater shared update structure under MOPD. Motivated by these findings, we propose geometry-based patching and base-preserving replay to improve OOD performance while largely preserving IND performance. Our findings identify a blind spot in capability consolidation via (M)OPD, highlighting the need for algorithms that explicitly support OOD retention and generalization.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.05609 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:15:49.002Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).