Metraipolis › archive › dc_7eb4429cd2db

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution — arXiv:2610.04845

Source: arXiv
Original: https://arxiv.org/abs/2610.04845
License (as archived): CC0-1.0
Archived: 2026-10-06T05:30:46.253Z
SHA-256 of the archived text: f248141b95e6b4a52d6b101a5df0249d763cac76c6c64ccbd7f03e200d4a4789
Stable URL of this page: https://metraipolis.com/documento/dc_7eb4429cd2db

How to cite

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution — arXiv:2610.04845 — arXiv, https://arxiv.org/abs/2610.04845 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7eb4429cd2db (Metraipolis archive dc_7eb4429cd2db, sha256 f248141b95e6b4a52d6b101a5df0249d763cac76c6c64ccbd7f03e200d4a4789)

BibTeX

@misc{dc7eb4429cd2db,
  title = {Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution — arXiv:2610.04845},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04845},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7eb4429cd2db; sha256 f248141b95e6b4a52d6b101a5df0249d763cac76c6c64ccbd7f03e200d4a4789},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04845. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.04845 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.728Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).