Metraipolis › archive › dc_8bc66aade359

Groupwise Distortion Guarantees for Preference-Based Alignment — arXiv:2610.05450

Source
arXiv
Original
https://arxiv.org/abs/2610.05450
License (as archived)
CC0-1.0
Archived
2026-10-06T05:31:06.187Z
SHA-256 of the archived text
52825107a54474d33f0258c9293994d136da594256b3e99513245d54117217b4
Stable URL of this page
https://metraipolis.com/documento/dc_8bc66aade359

How to cite

Groupwise Distortion Guarantees for Preference-Based Alignment — arXiv:2610.05450 — arXiv, https://arxiv.org/abs/2610.05450 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8bc66aade359 (Metraipolis archive dc_8bc66aade359, sha256 52825107a54474d33f0258c9293994d136da594256b3e99513245d54117217b4)
BibTeX
@misc{dc8bc66aade359,
  title = {Groupwise Distortion Guarantees for Preference-Based Alignment — arXiv:2610.05450},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05450},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8bc66aade359; sha256 52825107a54474d33f0258c9293994d136da594256b3e99513245d54117217b4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05450. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Preference-based alignment methods such as reinforcement learning from human feedback (RLHF) and Nash learning from human feedback (NLHF) aggregate pairwise preferences to learn an LLM policy, but a natural goal is maximizing social welfare (average cardinal utility), which comparisons alone need not identify. Gölz, Haghtalab, and Yang (GHY) measure the gap by distortion: the worst-case ratio between the welfare of the best fixed lottery (distribution over responses) and of the learned lottery. They show NLHF is optimal when every user receives the same lottery. Account-based LLMs, however, have information about their users and can serve different lotteries to different people. We give an efficient algorithm, GLHF, that learns a single group-conditioned policy from one comparison per user. Under individual Bradley--Terry comparisons, GLHF asymptotically matches GHY's optimal population distortion bound simultaneously on every group in a prespecified, possibly overlapping collection, with sample complexity growing logarithmically in the number of groups and inversely with the smallest group mass. A sharper guarantee for groups with similar preferences approaches distortion of one when members share a feasible favorite response. In experiments using human coffee ratings and synthetic LLM-generated ratings, GLHF lowers distortion in every evaluated group and substantially reduces worst-group distortion relative to NLHF and other group-agnostic baselines.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.05450 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.796Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).