Metraipolis › archive › dc_195240ce67e2

Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows — arXiv:2609.36812

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36812
License (as archived)
CC0-1.0
Archived
2026-10-07T16:26:24.303Z
SHA-256 of the archived text
04a03d8fc4190f821f55d9ab51bafc7f12c8246d0690f7f43562714538893c46
Stable URL of this page
https://metraipolis.com/documento/dc_195240ce67e2

How to cite

Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows — arXiv:2609.36812 — arXiv (en), https://arxiv.org/abs/2609.36812 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_195240ce67e2 (Metraipolis archive dc_195240ce67e2, sha256 04a03d8fc4190f821f55d9ab51bafc7f12c8246d0690f7f43562714538893c46)
BibTeX
@misc{dc195240ce67e2,
  title = {Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows — arXiv:2609.36812},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36812},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_195240ce67e2; sha256 04a03d8fc4190f821f55d9ab51bafc7f12c8246d0690f7f43562714538893c46},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36812. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diffusion and flow policies can model complex behaviors in offline reinforcement learning (RL). However, penalizing their KL divergence from the behavior policy can discourage actions having high critic values with low behavior density. Directly refining behavior proposals may be an alternative, yet Gaussian or deterministic editors limit expressiveness to represent multiple separated modes for the same proposal. In this work, we introduce Proposal-Conditioned Refinement Flows (PReFlow), a policy extraction method combining critic-based proposal selection with a conditional refinement flow. To optimize proposal selection and refinement together, we formulate a KL-regularized objective whose optimum induces a Gibbs policy over final actions under a Gaussian-smoothed behavior prior. The refinement flow can represent multiple high value modes, while a proposal-centered Gaussian reference regulates large action changes. This Gaussian reference further enables us to make use of simulation-free, closed form adjoint matching targets from sampled endpoints and critic gradients, yielding a single velocity regression loss without a backward adjoint solve. On 50 OGBench tasks, PReFlow achieves competitive offline performance and the highest aggregate score among the compared methods after online fine-tuning, reaching 91\% after 500K environment steps.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2609.36812 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.893Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).