Metraipolis › archive › dc_2b908ecec481

Latent Reward Registers for Diffusion Preference Alignment — arXiv:2608.03929

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03929
License (as archived)
CC0-1.0
Archived
2026-10-31T12:42:35.517Z
SHA-256 of the archived text
e84cdf497d08dcc2553c4844dbb1c0c5e07d8619e715dbd571159cdeb315fcbe
Stable URL of this page
https://metraipolis.com/documento/dc_2b908ecec481

How to cite

Latent Reward Registers for Diffusion Preference Alignment — arXiv:2608.03929 — arXiv (en), https://arxiv.org/abs/2608.03929 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2b908ecec481 (Metraipolis archive dc_2b908ecec481, sha256 e84cdf497d08dcc2553c4844dbb1c0c5e07d8619e715dbd571159cdeb315fcbe)
BibTeX
@misc{dc2b908ecec481,
  title = {Latent Reward Registers for Diffusion Preference Alignment — arXiv:2608.03929},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03929},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2b908ecec481; sha256 e84cdf497d08dcc2553c4844dbb1c0c5e07d8619e715dbd571159cdeb315fcbe},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03929. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, which creates a severe temporal credit-assignment problem across the denoising process. We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents. Learnable, position-free register tokens are appended as an auxiliary read path to a frozen Diffusion Transformer (DiT), extracting preference signals without altering the generator's hidden states or velocity field. The resulting dense, differentiable reward field spans the full denoising trajectory and supports two alignment strategies. For training, Reward-Gradient On-Policy Distillation (RG-OPD) converts this dense reward field into per-step targets at states visited by the current generator, replacing rollout-intensive policy gradients with direct on-policy distillation. For inference, Reward-Guided Sampling (RGS) steers trajectories with magnitude-matched reward-gradient corrections and no parameter updates. Empirically, at high noise levels (t=0.8) the registers reach the highest pairwise accuracy among the evaluated latent reward models. RG-OPD outperforms online reinforcement learning baselines while reducing GPU hours by up to 33x. RGS achieves significant reward improvement with a favorable reward-quality balance against training-free baselines. Code and weights are to be available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.03929 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:31:30.383Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).