Metraipolis › archive › dc_deb86a275f22

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback — arXiv:2611.07561

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.07561
License (as archived)
CC0-1.0
Archived
2026-11-06T14:50:34.528Z
SHA-256 of the archived text
76d5b723971d28977c05267f4536879df4221f1981c11c72d8ccdc6e34156801
Stable URL of this page
https://metraipolis.com/documento/dc_deb86a275f22

How to cite

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback — arXiv:2611.07561 — arXiv (en), https://arxiv.org/abs/2611.07561 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_deb86a275f22 (Metraipolis archive dc_deb86a275f22, sha256 76d5b723971d28977c05267f4536879df4221f1981c11c72d8ccdc6e34156801)
BibTeX
@misc{dcdeb86a275f22,
  title = {Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback — arXiv:2611.07561},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07561},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_deb86a275f22; sha256 76d5b723971d28977c05267f4536879df4221f1981c11c72d8ccdc6e34156801},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07561. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Safe reinforcement learning from human feedback (Safe-RLHF) seeks to improve helpfulness while satisfying harmlessness constraints. Existing primal-dual approaches can converge to feasible policies but do not prevent constraint violations during training. We study a stronger requirement, safe exploration for RLHF, under which every policy executed during training, including auxiliary exploration policies, remains feasible. We propose Zeroth-Order Log-Barrier Policy Gradient (ZO-LBPG), a reward-model-free method that optimizes a log-barrier objective using zeroth-order estimates derived directly from human preferences. To ensure safety under noisy feedback, we combine conservative constraint margin estimates with coupled rollouts and an adaptive step size that prevents updates from crossing constraint boundaries. We prove that, with high probability, all iterates and exploration policies remain strictly feasible, and that ZO-LBPG globally converges to an approximately optimal constrained policy. We establish a sample complexity of $\widetilde{\mathcal O}(\varepsilon^{-6}+d^3\varepsilon^{-5})$ trajectories and $\widetilde{\mathcal O}(\varepsilon^{-8}+d^5\varepsilon^{-7})$ human queries. Experiments on a synthetic CMDP and a language-model alignment task corroborate the training-time safety guarantees.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.07561 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:21:42.627Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).