Metraipolis › archive › dc_4e3478ad46cb

Diffusion Reward Models — arXiv:2609.33803

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33803
License (as archived)
CC0-1.0
Archived
2026-10-07T21:01:47.125Z
SHA-256 of the archived text
50719010802b887b3f8b0bd38973832a6e6eb6afdab37dd9d95fef415ee3c363
Stable URL of this page
https://metraipolis.com/documento/dc_4e3478ad46cb

How to cite

Diffusion Reward Models — arXiv:2609.33803 — arXiv (en), https://arxiv.org/abs/2609.33803 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e3478ad46cb (Metraipolis archive dc_4e3478ad46cb, sha256 50719010802b887b3f8b0bd38973832a6e6eb6afdab37dd9d95fef415ee3c363)
BibTeX
@misc{dc4e3478ad46cb,
  title = {Diffusion Reward Models — arXiv:2609.33803},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33803},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e3478ad46cb; sha256 50719010802b887b3f8b0bd38973832a6e6eb6afdab37dd9d95fef415ee3c363},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33803. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reward models underpin the alignment of large language models, yet the dominant designs reduce each prompt--response pair to a point estimate or to a distribution from a fixed parametric family. This is at odds with human preference, which is inherently multimodal: the same response can be reasonably judged in many ways, and no single family covers all of them. To better fit this structure, we introduce DRM, a Diffusion Reward Model that recasts reward modeling as conditional density estimation over $p(\mathbf{r}\mid x,y)$. Conditioned on a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector, placing no parametric assumption on the output distribution and naturally representing its multimodal structure. A single architecture handles both multi-attribute regression and pairwise preference data, and at inference $N$ samples form an empirical reward distribution that can be aggregated into a scalar, a variance, or quantiles. Across five benchmarks, DRM matches or surpasses baselines under matched data and backbone, stays competitive with much larger discriminative, distributional, and generative RMs despite its modest training scale, and recovers multimodal reward structure where conventional heads collapse to a point. Uncertainty-aware rejection and lower-confidence-bound (LCB) aggregation further demonstrate that DRM can exploit distributional information beyond a scalar reward to improve reward-model decisions. Downstream RLHF experiments additionally show that using DRM as the training-time reward leads to improved policy performance, directly validating the practical benefit of diffusion-based reward modeling for RLHF training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.33803 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.750Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).