Metraipolis › archive › dc_10ed30fc0d43
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution — arXiv:2610.06804
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.06804
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:27:53.027Z
- SHA-256 of the archived text
b9b1bf58d7451319fc9e6605065763ed57c78b4e35c137d6aeec2ae953b0ebe2- Stable URL of this page
https://metraipolis.com/documento/dc_10ed30fc0d43
How to cite
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution — arXiv:2610.06804 — arXiv, https://arxiv.org/abs/2610.06804 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_10ed30fc0d43 (Metraipolis archive dc_10ed30fc0d43, sha256 b9b1bf58d7451319fc9e6605065763ed57c78b4e35c137d6aeec2ae953b0ebe2)
BibTeX
@misc{dc10ed30fc0d43,
title = {Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution — arXiv:2610.06804},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.06804},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_10ed30fc0d43; sha256 b9b1bf58d7451319fc9e6605065763ed57c78b4e35c137d6aeec2ae953b0ebe2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06804. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A language model can give a correct answer more probability than any single incorrect answer and still usually sample an incorrect one, because the incorrect answers together hold more probability. The power distribution raises each complete answer's probability to a power above one and renormalizes, shifting probability toward answers the model finds most likely (sharpening). Sampling from it improves reasoning without changing parameters, but needs many scored candidates per query. We show that a model can instead be trained to produce such answers in one generation. On-policy power distillation (OPPD) runs a sequential Monte Carlo sampler in which the model being trained generates candidates and a frozen teacher's power distribution weights them; the same probabilities weight each answer in a maximum-likelihood update. Training raises single-generation accuracy by up to 23.0 points on MATH500 and 27.3 on GSM8K over the untrained model at the same temperature, and one generation scores 2.4 and 3.5 points above published power sampling with 64 candidates, recovering 94 percent of the gain that 16 candidates give the untrained model. For context, against GRPO trained with verified rewards from the same checkpoint and budget, OPPD scores 3.8, 4.0 and 5.4 points higher on MATH500, GSM8K and AIME using no reference answers; the two are complementary, and OPPD applied after GRPO adds up to 9.3 points. Trained only on mathematics, OPPD raises HumanEval accuracy by up to 5.3 points. One loss coefficient moves the sharpening exponent the model absorbs between 1.19 and 2.02, against 1.14 for ordinary on-policy distillation, and it rises mostly on the model's own answers. Gains hold across model families and sizes, including a model already trained with verified rewards, where lowering the temperature gives nothing and OPPD adds 4.4 points on MATH500. Code: this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.06804 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:06.192Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).