Metraipolis › archive › dc_dcb31737579e

Optimizing Stochastic Policy Geometry for Few-Step Flow Reinforcement Learning — arXiv:2610.41949

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41949
License (as archived)
CC0-1.0
Archived
2026-11-02T07:19:15.698Z
SHA-256 of the archived text
1f3a0aceb9f085857395f40b9e32d0fbd4fd6afd5bc7b188bfce8300c5409849
Stable URL of this page
https://metraipolis.com/documento/dc_dcb31737579e

How to cite

Optimizing Stochastic Policy Geometry for Few-Step Flow Reinforcement Learning — arXiv:2610.41949 — arXiv (en), https://arxiv.org/abs/2610.41949 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dcb31737579e (Metraipolis archive dc_dcb31737579e, sha256 1f3a0aceb9f085857395f40b9e32d0fbd4fd6afd5bc7b188bfce8300c5409849)
BibTeX
@misc{dcdcb31737579e,
  title = {Optimizing Stochastic Policy Geometry for Few-Step Flow Reinforcement Learning — arXiv:2610.41949},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41949},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dcb31737579e; sha256 1f3a0aceb9f085857395f40b9e32d0fbd4fd6afd5bc7b188bfce8300c5409849},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41949. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Few-step flow models generate samples through a handful of large denoising transitions. Applying GRPO to such models requires introducing stochasticity into the denoising transitions, and when a single transition spans a wide range of noise levels, the choice of the resulting stochastic policy becomes a major factor in optimization. Recent constructions constrain the policy to preserve the target flow coefficients, but this constraint admits a family of policies and offers no criterion for choosing among them. We introduce Geo-FSRL (Geometry-Optimized Few-Step Reinforcement Learning), built on a selection principle: at a fixed conditioning state and timestep pair, choose the admissible policy that minimizes the conditional KL divergence induced by any given nonzero velocity-output update. Within a coefficient-preserving family of isotropic Gaussian transitions, this criterion reduces to minimizing a single scalar geometry factor that determines the conditional KL for every velocity-output update. The same factor governs velocity-space Fisher information and conditional transition log-likelihood-ratio variance. We derive the unique closed-form optimizer within this family, shared across all nonzero update magnitudes and directions, with transition coefficients determined by both the source and target noise levels. Because the minimized geometry factor still varies across timesteps, we further adopt gradient-scale matching from prior work and derive closed-form policy-gradient and KL weights that normalize explicit score scales and reference-KL curvature. Geo-FSRL trains on the original few-step schedule and retains the deterministic solver at inference. Our method achieves higher final reward scores than Flow-GRPO and Flow-CPS on both image generation and video generation models under the 4-step setting. Code is available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.41949 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:16:57.201Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).