Metraipolis › archive › dc_df50685c50a5

Rethinking On-Policy Distillation of Large Language Models II: One Training Example — arXiv:2609.04172

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04172
License (as archived)
CC0-1.0
Archived
2026-10-17T01:11:36.809Z
SHA-256 of the archived text
c241cc88a924cddbe6a2767844295810bbf1ec4f66cd6893816c76ebff18e5a6
Stable URL of this page
https://metraipolis.com/documento/dc_df50685c50a5

How to cite

Rethinking On-Policy Distillation of Large Language Models II: One Training Example — arXiv:2609.04172 — arXiv (en), https://arxiv.org/abs/2609.04172 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_df50685c50a5 (Metraipolis archive dc_df50685c50a5, sha256 c241cc88a924cddbe6a2767844295810bbf1ec4f66cd6893816c76ebff18e5a6)
BibTeX
@misc{dcdf50685c50a5,
  title = {Rethinking On-Policy Distillation of Large Language Models II: One Training Example — arXiv:2609.04172},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04172},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_df50685c50a5; sha256 c241cc88a924cddbe6a2767844295810bbf1ec4f66cd6893816c76ebff18e5a6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04172. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure \emph{state coverage}, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches \(71.5\%\), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach \(98.9\%\) and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.04172 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T00:28:02.918Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).