Metraipolis › archive › dc_ca5ff8725c8d

Order-Only On-Policy Distillation — arXiv:2611.02112

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02112
License (as archived): CC0-1.0
Archived: 2026-11-03T18:24:38.991Z
SHA-256 of the archived text: 2157c1ac248d11da5760f97d06fdd5d073dc6baab087a0cdb2c2f8dd02e2f81d
Stable URL of this page: https://metraipolis.com/documento/dc_ca5ff8725c8d

How to cite

Order-Only On-Policy Distillation — arXiv:2611.02112 — arXiv (en), https://arxiv.org/abs/2611.02112 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca5ff8725c8d (Metraipolis archive dc_ca5ff8725c8d, sha256 2157c1ac248d11da5760f97d06fdd5d073dc6baab087a0cdb2c2f8dd02e2f81d)

BibTeX

@misc{dcca5ff8725c8d,
  title = {Order-Only On-Policy Distillation — arXiv:2611.02112},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02112},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca5ff8725c8d; sha256 2157c1ac248d11da5760f97d06fdd5d073dc6baab087a0cdb2c2f8dd02e2f81d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02112. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) trains a student on its own generations by matching the teacher's token probabilities. These probabilities entangle two kinds of information: which tokens the teacher prefers, and how confident it is. The latter reflects the teacher's own capacity and calibration, which a smaller student may be unable to reproduce. Because the teacher temperature changes these values without changing the ordering, OPD is also sensitive to the temperature choice. We ask whether the teacher's ordering of candidate tokens is enough, and propose Order-Only On-Policy Distillation (O$^{3}$PD). At each student-generated prefix, O$^{3}$PD trains the student to reproduce the order of the teacher's top-$K$ tokens, without fitting their probability values. The objective is therefore invariant to the teacher temperature and needs no temperature tuning. Across six teacher--student pairs, O$^{3}$PD outperforms OPD on the average over HMMT25, AIME24, and AIME25 in every pair, by up to 2.9 points, with gains of up to 5.6 points on AIME25. Other ranking losses yield similar gains, so the benefit comes from ordinal supervision itself. Analysis shows that O$^{3}$PD departs further from the teacher's probabilities than OPD, yet reproduces its ranking more faithfully and yields a stronger student. The ranking, not the probability values, thus carries the knowledge that on-policy distillation needs to transfer.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.02112 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:08.042Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).