Metraipolis › archive › dc_ce26df45ab59

Subspace Inference Enables Efficient Active Reward Learning from Preferences — arXiv:2609.04066

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04066
License (as archived)
CC0-1.0
Archived
2026-10-17T01:12:15.486Z
SHA-256 of the archived text
cdaef7b4f0bd747c1fc9cac46f56a9ac5f6e8abcda9dd423dd5934580cf0323b
Stable URL of this page
https://metraipolis.com/documento/dc_ce26df45ab59

How to cite

Subspace Inference Enables Efficient Active Reward Learning from Preferences — arXiv:2609.04066 — arXiv (en), https://arxiv.org/abs/2609.04066 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ce26df45ab59 (Metraipolis archive dc_ce26df45ab59, sha256 cdaef7b4f0bd747c1fc9cac46f56a9ac5f6e8abcda9dd423dd5934580cf0323b)
BibTeX
@misc{dcce26df45ab59,
  title = {Subspace Inference Enables Efficient Active Reward Learning from Preferences — arXiv:2609.04066},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04066},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ce26df45ab59; sha256 cdaef7b4f0bd747c1fc9cac46f56a9ac5f6e8abcda9dd423dd5934580cf0323b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04066. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning from human feedback (RLHF) has emerged as a powerful yet sample-inefficient approach for learning reward models from human preferences, making active learning a critical component in synthesizing informative preference queries. However, effective uncertainty quantification required for active learning remains a key challenge for large neural network reward models. In this paper, we introduce PreferenceEKF, a sample-efficient approach that tracks reward model uncertainty by framing active preference learning as a sequential Bayesian filtering problem. Instead of relying on computationally prohibitive posterior inference over the full neural network parameter space, our method performs sequential inference via an extended Kalman filter within a low-dimensional parameter subspace, continuously updating the reward model posterior as new preference queries arrive. Our approach enables scalable sampling of neural network parameters to efficiently compute acquisition functions for active reward learning. Experiments on the D4RL and V-D4RL benchmarks demonstrate that our approach achieves better sample efficiency, runtime, scalability, and calibration compared to other Bayesian deep learning approaches, and the learned reward models lead to competitive offline reinforcement learning policy performance. This highlights the potential of scalable Bayesian methods for preference-based reward modeling in RLHF. Our code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2609.04066 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T00:28:03.027Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).