Metraipolis › archive › dc_ff58356d51c4

A Better Spur Should Start From Each Objective — arXiv:2609.08211

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.08211
License (as archived)
CC0-1.0
Archived
2026-10-16T13:59:27.950Z
SHA-256 of the archived text
c6e04c795bfdfc18d54a518afb956d44b9e6c119895ac58c8e5b624bec6ab1de
Stable URL of this page
https://metraipolis.com/documento/dc_ff58356d51c4

How to cite

A Better Spur Should Start From Each Objective — arXiv:2609.08211 — arXiv (en), https://arxiv.org/abs/2609.08211 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff58356d51c4 (Metraipolis archive dc_ff58356d51c4, sha256 c6e04c795bfdfc18d54a518afb956d44b9e6c119895ac58c8e5b624bec6ab1de)
BibTeX
@misc{dcff58356d51c4,
  title = {A Better Spur Should Start From Each Objective — arXiv:2609.08211},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.08211},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff58356d51c4; sha256 c6e04c795bfdfc18d54a518afb956d44b9e6c119895ac58c8e5b624bec6ab1de},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08211. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Real-world Multi-Objective Reinforcement Learning (MORL) often suffers from sparse rewards, reward conflicts, and late-stage reward tug-of-war, causing traditional linear scalarization to experience severe metric oscillations. To address optimization conflicts among multiple objectives in real-world deployment scenarios, we propose Multi-Marginal Preference Optimization (MMPO), a fine-grained framework that intervenes at the data, gradient, and constraint levels rather than relying on coarse-grained global scalarization. Specifically, MMPO performs exposure debiasing to mitigate sparse and biased rewards, applies priority-aware orthogonal projection to decouple conflicting gradients, and introduces self-prompted gradient constraints to prevent dominant objectives from overwhelming weaker ones. Experiments on real-world e-commerce datasets show that MMPO improves training stability and consistently achieves better performance across conflicting metrics. Moreover, it generalizes robustly to broader tasks such as ToolRL and code generation, demonstrating its effectiveness as a practical paradigm for multi-objective alignment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.08211 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:43.261Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).