Metraipolis › archive › dc_2face527cd89

Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters — arXiv:2611.09634

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.09634
License (as archived): CC0-1.0
Archived: 2026-11-09T22:35:09.746Z
SHA-256 of the archived text: b9f288dbc06801833411d1684e4881b90eb64b8223b6dc1d8133261f60887aeb
Stable URL of this page: https://metraipolis.com/documento/dc_2face527cd89

How to cite

Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters — arXiv:2611.09634 — arXiv (en), https://arxiv.org/abs/2611.09634 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2face527cd89 (Metraipolis archive dc_2face527cd89, sha256 b9f288dbc06801833411d1684e4881b90eb64b8223b6dc1d8133261f60887aeb)

BibTeX

@misc{dc2face527cd89,
  title = {Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters — arXiv:2611.09634},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.09634},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2face527cd89; sha256 b9f288dbc06801833411d1684e4881b90eb64b8223b6dc1d8133261f60887aeb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09634. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language model deployment often requires different trade-offs between competing objectives as user preferences change, yet standard post-training methods typically produce a single policy for a fixed objective weighting. We introduce LambdaMix, a new parameter efficient method for learning a preference conditioned family of policies with group-relative online reinforcement learning. LambdaMix jointly trains one low-rank adapter per objective over a frozen base model. One preference vector selects the adapter mixture that generates responses and weights the separately normalized rewards used to update the adapters. A single training run supports changing preferences at inference without further optimization. We experiment with four models on helpfulness and harmlessness and on solving math problems while following instructions. We also evaluate the two largest models on rewriting comments to preserve wording while reducing clues about personal attributes. On the first two tasks, LambdaMix achieves higher hypervolume than all evaluated conditioned and policy merging baselines across all four models. LambdaMix uses fewer training completions and stores fewer adapter parameters than a set of GRPO specialist policies. With architecture and initialization fixed, varying adapter mixtures and objective weights together gives wider score ranges and higher coverage than varying either alone.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.09634 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.337Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).