Metraipolis › archive › dc_5ccd6affba4d

What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection — arXiv:2609.05198

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.05198
License (as archived)
CC0-1.0
Archived
2026-10-16T22:21:16.910Z
SHA-256 of the archived text
9b6616cd65b29d73ef8cd917702145e29a088c3265bda5edd8ebe8b12b8d23ce
Stable URL of this page
https://metraipolis.com/documento/dc_5ccd6affba4d

How to cite

What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection — arXiv:2609.05198 — arXiv (en), https://arxiv.org/abs/2609.05198 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5ccd6affba4d (Metraipolis archive dc_5ccd6affba4d, sha256 9b6616cd65b29d73ef8cd917702145e29a088c3265bda5edd8ebe8b12b8d23ce)
BibTeX
@misc{dc5ccd6affba4d,
  title = {What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection — arXiv:2609.05198},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.05198},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5ccd6affba4d; sha256 9b6616cd65b29d73ef8cd917702145e29a088c3265bda5edd8ebe8b12b8d23ce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05198. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents an empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely 1-shot OPD. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model's improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problem naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., "Alternatively"). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even "unsolvable" examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on five models ranging from 1.5B to 8B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline. Code and models will be publicly available.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.05198 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:18:11.442Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).