Metraipolis › archive › dc_d112e094c2c4

Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning — arXiv:2610.04696

Source: arXiv
Original: https://arxiv.org/abs/2610.04696
License (as archived): CC0-1.0
Archived: 2026-10-06T05:30:41.715Z
SHA-256 of the archived text: 139ef49e5474c6d67647ff5b66e127b42a5e96fb781b6ba57586fd249192e420
Stable URL of this page: https://metraipolis.com/documento/dc_d112e094c2c4

How to cite

Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning — arXiv:2610.04696 — arXiv, https://arxiv.org/abs/2610.04696 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d112e094c2c4 (Metraipolis archive dc_d112e094c2c4, sha256 139ef49e5474c6d67647ff5b66e127b42a5e96fb781b6ba57586fd249192e420)

BibTeX

@misc{dcd112e094c2c4,
  title = {Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning — arXiv:2610.04696},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04696},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d112e094c2c4; sha256 139ef49e5474c6d67647ff5b66e127b42a5e96fb781b6ba57586fd249192e420},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04696. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diffusion and flow models provide expressive policy classes for online reinforcement learning (RL), enabling multimodal behaviors and improved performance. However, training these policies remains challenging: the critic specifies the desired policy as an unnormalized Boltzmann density but does not provide direct samples from it. Many existing methods rely on importance sampling to construct training signals, which can suffer from high variance, increasing computational cost and destabilizing training. We propose Score-Calibrated Flow (SCF), a simple and efficient algorithm for training generative models to sample from unnormalized densities without importance sampling or backpropagation through the sampling trajectory. We learn the desired flow by enforcing self-consistency, bypassing target posterior mean estimation. By jointly exploiting the prescribed target score and the structure of flow matching, we establish these self-consistency requirements as score-calibrated optimality conditions, first for the terminal density and then for the trainable velocity field. We prove that their unique solutions are, respectively, the target density and the ideal flow model that conditional flow matching (CFM) would recover if target samples were available. We formulate the velocity condition as a fixed-point equation and exploit its conditional-expectation structure to construct a stop-gradient objective for enforcing it. The resulting training procedure retains the scalable sample-interpolate-regress structure of CFM despite the absence of target samples, using endpoints generated by the current flow. For online RL, the critic gradient supplies the target score at the generated actions, yielding a direct approach to actor training. Experiments on RL benchmarks demonstrate that SCF matches or improves upon state-of-the-art generative-policy baselines, while substantially reducing training time.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Systems and Control (eess.SY).
Pagina del lavoro: https://arxiv.org/abs/2610.04696 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.710Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).