Metraipolis › archive › dc_df3d7f4fcc07

Self-Privileged Fine-Tuning — arXiv:2611.04632

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04632
License (as archived)
CC0-1.0
Archived
2026-11-04T12:33:42.557Z
SHA-256 of the archived text
80e45ec5424d270ab5934fe96850c2c4ad73e9d4d574f8a0d3cff76fc44a21fd
Stable URL of this page
https://metraipolis.com/documento/dc_df3d7f4fcc07

How to cite

Self-Privileged Fine-Tuning — arXiv:2611.04632 — arXiv (en), https://arxiv.org/abs/2611.04632 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_df3d7f4fcc07 (Metraipolis archive dc_df3d7f4fcc07, sha256 80e45ec5424d270ab5934fe96850c2c4ad73e9d4d574f8a0d3cff76fc44a21fd)
BibTeX
@misc{dcdf3d7f4fcc07,
  title = {Self-Privileged Fine-Tuning — arXiv:2611.04632},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04632},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_df3d7f4fcc07; sha256 80e45ec5424d270ab5934fe96850c2c4ad73e9d4d574f8a0d3cff76fc44a21fd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04632. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Supervised fine-tuning (SFT) is a key component of language model post-training, enabling effective task adaptation and serving as an important foundation for subsequent reinforcement learning. Recent methods seek to improve SFT by introducing token-level weights into the training objective, where these weights are typically derived from a model's output distribution at each token position. To better understand how such weighting schemes affect SFT, we analyze their optimization objectives and theoretically characterize the target distributions they induce. Our analysis shows that dynamic weighting methods, such as Dynamic Fine-Tuning (DFT), induce a heavily sharpened distribution, whereas static weighting methods fit a target distribution proportional to the product of the data distribution and a frozen prior. These findings suggest that static weighting provides a more stable training target, while its effectiveness critically depends on the quality of the prior distribution. Building on this observation, we propose Self-Privileged Fine-Tuning (SPFT), which uses a frozen copy of the initial model conditioned on privileged information (such as ground truth for mathematical questions) as a prior to provide guidance for token-level weighting during fine-tuning. Through a Bayesian decomposition, we further show that SPFT reweights the standard static prior according to how informative each target token is for predicting the ground-truth outcome, thereby assigning larger weights to more predictive tokens. Experiments on mathematical reasoning and code generation show that SPFT improves over standard SFT and other baselines across the main settings.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04632 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:27:02.073Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).