Metraipolis › archive › dc_ccff161ee942
Value Forcing: Aligning Return Distributions at All Noise Levels — arXiv:2611.02160
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02160
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:19:44.024Z
- SHA-256 of the archived text
cc4a8fee335a1a40510d9defc88d55e2b449e0038dfbce038360e973e24f79e6- Stable URL of this page
https://metraipolis.com/documento/dc_ccff161ee942
How to cite
Value Forcing: Aligning Return Distributions at All Noise Levels — arXiv:2611.02160 — arXiv (en), https://arxiv.org/abs/2611.02160 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ccff161ee942 (Metraipolis archive dc_ccff161ee942, sha256 cc4a8fee335a1a40510d9defc88d55e2b449e0038dfbce038360e973e24f79e6)
BibTeX
@misc{dcccff161ee942,
title = {Value Forcing: Aligning Return Distributions at All Noise Levels — arXiv:2611.02160},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02160},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ccff161ee942; sha256 cc4a8fee335a1a40510d9defc88d55e2b449e0038dfbce038360e973e24f79e6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02160. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Learning accurate value functions is central to reinforcement learning (RL), yet remains challenging because supervision is limited and indirect: ground-truth values are unavailable, and each transition in the data only provides an immediate reward that can be used to construct bootstrapped targets as supervision. This challenge is particularly pronounced when data is limited or reward signals are sparse, as is common in real-world applications such as robotics. We argue that the key to accurate value learning in this limited data regime is extracting richer learning signals from the available data. In this paper, we introduce Value Forcing, a new approach that uses flow matching to estimate return distributions and enables accurate value learning from limited data. Our key insight is that we can extract richer learning signals by enforcing Bellman consistency at every noise level. Empirically, Value Forcing improves the accuracy of estimating the return distribution, and prevents the distribution from collapsing to a single value, as occurs with prior methods. We instantiate Value Forcing with two practical value-learning algorithms: VF, an actor-critic method, and IVF, an implicit value-learning method. We evaluate both algorithms across $26$ long-horizon tasks in the offline RL setting and find that they outperform prior value-learning methods, with particularly large gains when data is limited. Our project website is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.02160 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.661Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).