Metraipolis › archive › dc_2e3a9c8bf290
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner — arXiv:2609.12651
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.12651
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:09:07.971Z
- SHA-256 of the archived text
df6ff5c5796260329a23579ae82de04c810aab55bd837f177ab1a2c9327a819a- Stable URL of this page
https://metraipolis.com/documento/dc_2e3a9c8bf290
How to cite
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner — arXiv:2609.12651 — arXiv (en), https://arxiv.org/abs/2609.12651 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2e3a9c8bf290 (Metraipolis archive dc_2e3a9c8bf290, sha256 df6ff5c5796260329a23579ae82de04c810aab55bd837f177ab1a2c9327a819a)
BibTeX
@misc{dc2e3a9c8bf290,
title = {Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner — arXiv:2609.12651},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.12651},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2e3a9c8bf290; sha256 df6ff5c5796260329a23579ae82de04c810aab55bd837f177ab1a2c9327a819a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12651. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gölz et al. (2025) demonstrated that the \textit{distortion} -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter $\beta$ when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data ($\mu$) and the KL reference policy ($\pi_{\mathrm{ref}}$). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is $\tilde{\Theta}(\beta B + \beta)$, where $B$ is an upper bound on the log density ratio between $\mu$ and $\pi_{\mathrm{ref}}$. In particular, when there is no distribution mismatch (i.e., $\mu = \pi_{\mathrm{ref}}$), RLHF achieves the optimal distortion of $O(\beta)$ up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to $\mu$.
Tema degli esploratori: economia dei servizi IA (retro-catalogo arXiv: annunci del 2026-09-14 in cs.GT).
Categorie arXiv: Machine Learning (cs.LG) ; Computer Science and Game Theory (cs.GT).
Pagina del lavoro: https://arxiv.org/abs/2609.12651 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:04:07.422Z dalla pagina https://arxiv.org/catchup/cs.GT/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).