Metraipolis › archive › dc_99e8be0aa297
On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training — arXiv:2609.36659
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36659
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:29:17.779Z
- SHA-256 of the archived text
cc986795ade34503287b46ab0024b4533ac7ae32c48c7909157bf7a32799b5e8- Stable URL of this page
https://metraipolis.com/documento/dc_99e8be0aa297
How to cite
On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training — arXiv:2609.36659 — arXiv (en), https://arxiv.org/abs/2609.36659 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_99e8be0aa297 (Metraipolis archive dc_99e8be0aa297, sha256 cc986795ade34503287b46ab0024b4533ac7ae32c48c7909157bf7a32799b5e8)
BibTeX
@misc{dc99e8be0aa297,
title = {On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training — arXiv:2609.36659},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36659},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_99e8be0aa297; sha256 cc986795ade34503287b46ab0024b4533ac7ae32c48c7909157bf7a32799b5e8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36659. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization principles for improving the generalization of other paradigms such as supervised fine-tuning (SFT). To address this limitation, we investigate whether there exists a specific on-policy update behavior that can achieve such improvements. First, our analyses reveal that SFT updates parameters along consistent directions, while the on-policy paradigm continuously adjusts the direction during training. This difference inspires us to focus on the cumulative update direction of each parameter as a promising behavior. Then, we evaluate its effectiveness for improving generalization by proposing On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the direction identified by on-policy paradigms. The strong performance of OPSFT indicates that the generalization advantage of on-policy paradigms can be transferred to SFT through the parameter update direction. Once such a direction is identified, even SFT can generalize with its updates constrained to this direction. This finding offers two practical benefits by combining the strong generalization of on-policy paradigms with the advantages of SFT, including the high training efficiency and ability to leverage high-quality trajectories. For efficiency, we identify update directions that support strong generalization using a few on-policy training steps, and subsequently apply OPSFT to achieve high training efficiency. For leveraging high-quality trajectories, OPSFT can utilize these trajectories to continue improving a post-trained model along its update direction without disrupting the ability learned from on-policy training. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.36659 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.207Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).