Metraipolis › archive › dc_24950c85fded
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO — arXiv:2611.08453
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.08453
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-06T14:43:24.440Z
- SHA-256 of the archived text
26ec30f4c1f45926c28ded2182b1ad97d24fc1f05e16242c12e28acc8c27e93b- Stable URL of this page
https://metraipolis.com/documento/dc_24950c85fded
How to cite
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO — arXiv:2611.08453 — arXiv (en), https://arxiv.org/abs/2611.08453 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_24950c85fded (Metraipolis archive dc_24950c85fded, sha256 26ec30f4c1f45926c28ded2182b1ad97d24fc1f05e16242c12e28acc8c27e93b)
BibTeX
@misc{dc24950c85fded,
title = {Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO — arXiv:2611.08453},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.08453},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_24950c85fded; sha256 26ec30f4c1f45926c28ded2182b1ad97d24fc1f05e16242c12e28acc8c27e93b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08453. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with $50\times$ fewer distillation FLOPs than GRPO needs to reach the same accuracy. Following the same teacher setting, we consistently observe this high efficiency with a larger student, on code, and in a second model family. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD (DeepSeek-AI, 2026) consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.08453 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-06T14:18:57.600Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).