Metraipolis › archive › dc_461967e76cf1

Decoupling Policy Extraction for Offline Reinforcement Learning — arXiv:2608.20909

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.20909
License (as archived)
CC0-1.0
Archived
2026-10-18T10:31:07.924Z
SHA-256 of the archived text
e60e34e5012e68e9912b920e7ce63b347139fd30b09ab0535bbe9614874e2ed1
Stable URL of this page
https://metraipolis.com/documento/dc_461967e76cf1

How to cite

Decoupling Policy Extraction for Offline Reinforcement Learning — arXiv:2608.20909 — arXiv (en), https://arxiv.org/abs/2608.20909 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_461967e76cf1 (Metraipolis archive dc_461967e76cf1, sha256 e60e34e5012e68e9912b920e7ce63b347139fd30b09ab0535bbe9614874e2ed1)
BibTeX
@misc{dc461967e76cf1,
  title = {Decoupling Policy Extraction for Offline Reinforcement Learning — arXiv:2608.20909},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.20909},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_461967e76cf1; sha256 e60e34e5012e68e9912b920e7ce63b347139fd30b09ab0535bbe9614874e2ed1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.20909. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-24 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2608.20909 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T10:27:55.943Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).