Metraipolis › archive › dc_ccb06e95b6fc

Refinement-Based Flow Policy Optimization — arXiv:2609.15123

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15123
License (as archived)
CC0-1.0
Archived
2026-10-15T22:53:39.466Z
SHA-256 of the archived text
8eafd0d346641f720ee6d6ca9d7c7a771a4ce722e11ccd09af73d9b2da89720b
Stable URL of this page
https://metraipolis.com/documento/dc_ccb06e95b6fc

How to cite

Refinement-Based Flow Policy Optimization — arXiv:2609.15123 — arXiv (en), https://arxiv.org/abs/2609.15123 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ccb06e95b6fc (Metraipolis archive dc_ccb06e95b6fc, sha256 8eafd0d346641f720ee6d6ca9d7c7a771a4ce722e11ccd09af73d9b2da89720b)
BibTeX
@misc{dcccb06e95b6fc,
  title = {Refinement-Based Flow Policy Optimization — arXiv:2609.15123},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15123},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ccb06e95b6fc; sha256 8eafd0d346641f720ee6d6ca9d7c7a771a4ce722e11ccd09af73d9b2da89720b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15123. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Flow-based policies offer an expressive representation for online reinforcement learning, but conventional flow matching requires samples drawn from the distribution to be modeled. This poses a challenge when the desired action distribution is defined only implicitly by a Q-function, since directly sampling actions from the resulting distribution is generally intractable. We propose Refinement-Based Flow Policy Optimization (RFPO), a novel framework for training a flow policy in online reinforcement learning by alternating between Q-guided sample refinement and self-target flow matching. RFPO first generates actions from Gaussian noise using the current flow policy and then uses a finite-step stochastic refinement procedure to move them toward an energy-based distribution induced by the Q-function. Each refined action is then paired with its corresponding initial noise sample and used as a fixed target for flow-matching training. By repeatedly refining its own outputs and learning from the resulting targets, RFPO incorporates Q-guidance into the policy without requiring direct samples from the target distribution, while retaining the capacity to represent multiple action modes. We further provide a theoretical analysis of the distributional dynamics induced by RFPO. Across six continuous-control tasks, RFPO matches or outperforms a standard Gaussian-policy baseline on almost every task. Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that RFPO captures complex multimodal structure without mode collapse.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.15123 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.812Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).