Metraipolis › archive › dc_9fd9a2ca01bc

On KL-Regularized Policy Optimization — arXiv:2610.08963

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08963
License (as archived)
CC0-1.0
Archived
2026-10-08T11:10:24.857Z
SHA-256 of the archived text
9fa1635765a6e965983b86d55c6b2cccd05adeee8f1d97014ae3497ab5a626bb
Stable URL of this page
https://metraipolis.com/documento/dc_9fd9a2ca01bc

How to cite

On KL-Regularized Policy Optimization — arXiv:2610.08963 — arXiv (en), https://arxiv.org/abs/2610.08963 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9fd9a2ca01bc (Metraipolis archive dc_9fd9a2ca01bc, sha256 9fa1635765a6e965983b86d55c6b2cccd05adeee8f1d97014ae3497ab5a626bb)
BibTeX
@misc{dc9fd9a2ca01bc,
  title = {On KL-Regularized Policy Optimization — arXiv:2610.08963},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08963},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9fd9a2ca01bc; sha256 9fa1635765a6e965983b86d55c6b2cccd05adeee8f1d97014ae3497ab5a626bb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08963. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-$K$ and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.08963 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:06.034Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).