Metraipolis › archive › dc_dc4d2ffa15fb

FERPO: Forward Entropy-Regularized Policy Optimization — arXiv:2610.02198

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.02198
License (as archived): CC0-1.0
Archived: 2026-10-06T12:16:32.857Z
SHA-256 of the archived text: b190cd8428a5d2f8cc709b87a06c08eb706541516bf761b12836f0304c336124
Stable URL of this page: https://metraipolis.com/documento/dc_dc4d2ffa15fb

How to cite

FERPO: Forward Entropy-Regularized Policy Optimization — arXiv:2610.02198 — arXiv (en), https://arxiv.org/abs/2610.02198 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc4d2ffa15fb (Metraipolis archive dc_dc4d2ffa15fb, sha256 b190cd8428a5d2f8cc709b87a06c08eb706541516bf761b12836f0304c336124)

BibTeX

@misc{dcdc4d2ffa15fb,
  title = {FERPO: Forward Entropy-Regularized Policy Optimization — arXiv:2610.02198},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.02198},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc4d2ffa15fb; sha256 b190cd8428a5d2f8cc709b87a06c08eb706541516bf761b12836f0304c336124},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.02198. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Robotics (cs.RO); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.02198 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.261Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).