Metraipolis › archive › dc_7697d7256069
To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization — arXiv:2608.18770
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.18770
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T13:53:25.204Z
- SHA-256 of the archived text
62d5e5d85e4d92c91f707e0975730084dc946c23e5c6385d36447ddd1b52d1b3- Stable URL of this page
https://metraipolis.com/documento/dc_7697d7256069
How to cite
To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization — arXiv:2608.18770 — arXiv (en), https://arxiv.org/abs/2608.18770 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7697d7256069 (Metraipolis archive dc_7697d7256069, sha256 62d5e5d85e4d92c91f707e0975730084dc946c23e5c6385d36447ddd1b52d1b3)
BibTeX
@misc{dc7697d7256069,
title = {To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization — arXiv:2608.18770},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.18770},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7697d7256069; sha256 62d5e5d85e4d92c91f707e0975730084dc946c23e5c6385d36447ddd1b52d1b3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18770. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Learning a reward model from human feedback and optimizing a policy against it is one approach to aligning AI systems with individual users. From a fairness perspective, existing work improves such alignment by developing data-efficient and accurate reward models that capture minority preferences despite scarce data. We push this line of inquiry one step further and argue that data-efficient and accurate per-user reward models are not sufficient: users whose reward models are difficult to \textit{optimize} at the policy level can become a new underserved group. We start from the observation that one user's reward model can be easy to optimize from the initial policy while another's is not. We argue that, given a sufficiently diverse user population, a curriculum naturally emerges between easy- and hard-to-optimize reward models. Building on this insight, we propose CurriPO, which grows a tree-structured curriculum to accommodate diverse user-specific objectives, covering the population in a single traversal. Specifically, CurriPO automatically constructs a curriculum over diverse user reward models, allowing it to branch from the existing curriculum and reuse reward models previously incorporated into the curriculum. To the best of our knowledge, this is the first work to explicitly exploit multi-user structure to address optimization in AI alignment. Extensive experiments on personalized continuous control in a simulated environment show that CurriPO achieves $1.2$--$2.1\times$ the population satisfaction of the strongest baseline while substantially reducing training time. Additional analysis attributes much of this improvement to the users left underserved by conventional optimization.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2608.18770 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:50:58.936Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).