Metraipolis › archive › dc_e7acd088276d
Specifying Reward Functions for RL Without Environment Sampling — arXiv:2609.15544
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15544
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T22:53:53.013Z
- SHA-256 of the archived text
ff76db4a6ae8e7b842624d4a141c69ea10ea272f501e25f84298f867ccb1f672- Stable URL of this page
https://metraipolis.com/documento/dc_e7acd088276d
How to cite
Specifying Reward Functions for RL Without Environment Sampling — arXiv:2609.15544 — arXiv (en), https://arxiv.org/abs/2609.15544 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e7acd088276d (Metraipolis archive dc_e7acd088276d, sha256 ff76db4a6ae8e7b842624d4a141c69ea10ea272f501e25f84298f867ccb1f672)
BibTeX
@misc{dce7acd088276d,
title = {Specifying Reward Functions for RL Without Environment Sampling — arXiv:2609.15544},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15544},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e7acd088276d; sha256 ff76db4a6ae8e7b842624d4a141c69ea10ea272f501e25f84298f867ccb1f672},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15544. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Enabling human stakeholders to specify reward functions that lead to their desired outcomes is a key challenge in deploying reinforcement learning agents. Preference-based methods such as online RLHF can reduce the burden of manual reward design, but they require repeatedly training policies, sampling trajectories from the real world, and eliciting feedback, making them impractical in settings where environment interaction is computationally expensive or unsafe. We introduce Experience-Free Autonomous Reward Specification (EARS), a method for learning reward functions from preferences without environment interaction. Our approach uses a structured LLM-mediated process to construct a small set of expressive reward features from a task description and the environment observation space, then strategically samples imagined trajectories in this feature space and learns feature weights from preferences over the imagined trajectory pairs. We evaluate on three long-horizon domains: pandemic lockdown regulation design, insulin administration for diabetes patients, and autonomous vehicle control on a highway. We compare EARS to baselines that also enable reward specification without environment interaction--namely, methods that directly prompt an LLM to generate a reward function. When learning from either ground-truth preference labels or preferences labeled by a LLM, EARS designs reward functions that are more aligned with the ground truth reward function that produced the preferences or LLM context than these baselines. These results suggest that preference-based reward specification remains effective without environment sampling, enabling practical reward design in settings where collecting real trajectories is costly or infeasible. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.15544 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:29:03.871Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).