Metraipolis › archive › dc_208a0d9797ef

Baselines and Score Centering in Policy Gradient Estimation — arXiv:2610.22263

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22263
License (as archived)
CC0-1.0
Archived
2026-10-16T10:24:52.174Z
SHA-256 of the archived text
67dd8a05f6bcdda49d49f5773b2871ca611138b326d70fd54bd0cd3301cd11ba
Stable URL of this page
https://metraipolis.com/documento/dc_208a0d9797ef

How to cite

Baselines and Score Centering in Policy Gradient Estimation — arXiv:2610.22263 — arXiv (en), https://arxiv.org/abs/2610.22263 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_208a0d9797ef (Metraipolis archive dc_208a0d9797ef, sha256 67dd8a05f6bcdda49d49f5773b2871ca611138b326d70fd54bd0cd3301cd11ba)
BibTeX
@misc{dc208a0d9797ef,
  title = {Baselines and Score Centering in Policy Gradient Estimation — arXiv:2610.22263},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22263},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_208a0d9797ef; sha256 67dd8a05f6bcdda49d49f5773b2871ca611138b326d70fd54bd0cd3301cd11ba},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22263. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning has become a widely used approach to post-training large language models. In a typical implementation, learning is off-policy: The policy generating the responses differs from the policy being trained because of training-inference mismatch or policy staleness in asynchronous training. This discrepancy is commonly addressed by importance sampling, whose ratio is usually regularized in practice through truncation, masking, or clipping. Under regularized importance sampling, the importance-weighted policy gradient is no longer invariant to constant shifts in the reward. This invariance is recovered by using the covariance between the reward and the importance-weighted score as the estimand, which coincides with the policy gradient when the ratio is not regularized. This note develops this covariance perspective for both sequence-level and token-level importance sampling. The covariance perspective reveals the roles of the group-mean baseline and score centering in policy gradient estimation. Both perform centering in covariance estimation, but they differ in the mean being subtracted: The group-mean baseline uses a sample mean, whereas score centering uses a population mean. The two lead to distinct unbiased covariance estimators with different variances. Whether both forms of centering should be used simultaneously depends on whether importance sampling is applied at the sequence level or at the token level. With sequence-level importance sampling, score centering adds no benefit once the group-mean baseline is used; with token-level importance sampling, using the group-mean baseline without score centering generally introduces bias when the importance ratio is regularized.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.22263 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:48:02.810Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).