Metraipolis › archive › dc_581bf8077c0e

A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation — arXiv:2609.22109

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.22109
License (as archived)
CC0-1.0
Archived
2026-10-09T07:05:57.862Z
SHA-256 of the archived text
cc9c442c23a490fe680708212afa77d3daf955ee9e3e3fd4dbde17080e943757
Stable URL of this page
https://metraipolis.com/documento/dc_581bf8077c0e

How to cite

A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation — arXiv:2609.22109 — arXiv (en), https://arxiv.org/abs/2609.22109 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_581bf8077c0e (Metraipolis archive dc_581bf8077c0e, sha256 cc9c442c23a490fe680708212afa77d3daf955ee9e3e3fd4dbde17080e943757)
BibTeX
@misc{dc581bf8077c0e,
  title = {A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation — arXiv:2609.22109},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22109},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_581bf8077c0e; sha256 cc9c442c23a490fe680708212afa77d3daf955ee9e3e3fd4dbde17080e943757},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22109. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Selective on-policy distillation trains a student only at the token positions a selector scores highest, and the literature compares selectors under a single shared learning rate--a control chosen to be neutral. We show it is not. Under LoRA on GSM8K (Qwen2.5-1.5B student, 7B teacher), across an 8x learning-rate grid, dense supervision is statistically flat (swing 1.8 pp, p=0.26) while every selective arm moves with the rate: 5.4 pp for a random 5% subset, 6.7 pp for a total-variation selector, up to 17.7 pp for a teachability selector. Consequently the dense-versus-selective verdict reads 10.1 pp at lr=1e-4 but 5.1 pp at 5e-5--a 2.0x difference decided by a parameter the protocol treats as scenery--and two of six pairwise significance calls between selectors flip between adjacent rates without any rank inversion. We call this selector-rate entanglement and trace it to selection itself rather than step size: AdamW update magnitudes track the rate to within 2.2% despite 15.5x gradient-norm differences across arms. A preregistered frozen-scoring ablation (selection scored by the initial student; criterion, budget, and on-policy rollouts unchanged; 12 seeds per cell) shows live scoring adds 3.79+/-1.69 pp of rate sensitivity (p=0.035) while the frozen arm remains significantly entangled (p=0.015): the feedback loop aggravates the phenomenon rather than causing it. Under full fine-tuning at the rates this literature actually uses (1e-6 to 1e-5) the pattern grows: dense itself swings 19.8 pp, the selective arm 49.5 pp, and the verdict ranges from a non-significant +3.6 pp at the published operating point to +34 pp (p=0.005) one notch hotter. On MATH-500 the rate dependence does not reproduce under LoRA, scoping that result, while the ~10 pp cost of selective training does. We prescribe reporting the arm x rate matrix, not a shared-rate column, as a precondition for selector comparisons.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.22109 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.113Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).