Metraipolis › archive › dc_bf8be0fc6d4d

On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning — arXiv:2609.26918

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.26918
License (as archived)
CC0-1.0
Archived
2026-10-08T09:12:00.856Z
SHA-256 of the archived text
9599aaa05711c43fa7650a1fd3c88d249c897e27cb0789f79a0454bba2a96a7e
Stable URL of this page
https://metraipolis.com/documento/dc_bf8be0fc6d4d

How to cite

On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning — arXiv:2609.26918 — arXiv (en), https://arxiv.org/abs/2609.26918 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf8be0fc6d4d (Metraipolis archive dc_bf8be0fc6d4d, sha256 9599aaa05711c43fa7650a1fd3c88d249c897e27cb0789f79a0454bba2a96a7e)
BibTeX
@misc{dcbf8be0fc6d4d,
  title = {On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning — arXiv:2609.26918},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.26918},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf8be0fc6d4d; sha256 9599aaa05711c43fa7650a1fd3c88d249c897e27cb0789f79a0454bba2a96a7e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26918. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hindsight relabeling which retroactively replacing a transition's goal with the outcome the agent actually achieved is an effective tool for improving sample-efficiency in Reinforcement Learning (RL). A natural extension to preference-conditioned multi-objective RL (MORL) relabels transitions with the preference direction the agent achieved rather than the one asked for. We show that this extension is frequently harmful: across four preference-conditioned off-policy algorithms spanning two critic backbones and two preference-sampling schemes on the continuous-control MO-Gymnasium suite, it degrades 19 of 36 algorithm-environment settings by as much as four standard deviations, improves only one, and leaves the rest unaffected. The harm is not a symptom of noisy relabels; denoising the target recovers almost nothing, and neither prioritized sampling nor any buffer-structural choice reproduces it. Instead, repeated relabeling collapses the critic's coverage onto whatever narrow region of the preference space the agent happened to visit. We name this failure mode \emph{Preference Coverage Collapse}, and quantify it with abandoned preference mass (APM), a value-aware statistic that tracks the harm ($\rho = -0.73$) where a purely structural coverage count does not. We then introduce \texttt{her\_mix}, a single-parameter convex combination pulling the achieved direction back towards the requested preference. At one fixed value across every algorithm and environment, it returns 16 of the 19 harmed settings to baseline, preserves and even improves the one setting in which relabeling helps, and cuts abandoned preference mass from $69\%$ to $6\%$. Protecting coverage over the preference simplex, not filtering noisy relabels, is what makes hindsight relabeling safe for MORL.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.26918 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T08:13:26.595Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).