Metraipolis › archive › dc_6be70ebb7162
Revisiting Critic Estimation in Soft Actor-Critic Algorithms — arXiv:2611.05666
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.05666
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-05T18:22:01.014Z
- SHA-256 of the archived text
25ec2de2ea6cedab8789e314efec28e05bc51962e8fc9f4ff8bb964030cb1fd0- Stable URL of this page
https://metraipolis.com/documento/dc_6be70ebb7162
How to cite
Revisiting Critic Estimation in Soft Actor-Critic Algorithms — arXiv:2611.05666 — arXiv (en), https://arxiv.org/abs/2611.05666 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6be70ebb7162 (Metraipolis archive dc_6be70ebb7162, sha256 25ec2de2ea6cedab8789e314efec28e05bc51962e8fc9f4ff8bb964030cb1fd0)
BibTeX
@misc{dc6be70ebb7162,
title = {Revisiting Critic Estimation in Soft Actor-Critic Algorithms — arXiv:2611.05666},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.05666},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6be70ebb7162; sha256 25ec2de2ea6cedab8789e314efec28e05bc51962e8fc9f4ff8bb964030cb1fd0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05666. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Soft Actor-Critic (SAC) is a widely used deep reinforcement learning algorithm that combines entropy regularization with an actor-critic architecture. Standard implementations rely on one-step temporal-difference learning of Q-functions, yet alternative critic design choices remain underexplored. In this work, we revisit three such choices: learning soft advantages instead of soft Q-values, the use of multi-step return targets, and the training of multi-step critics from replay buffer data. Keeping the policy objective fixed, we compare the six critic variants these choices generate: (i) advantage and (ii) dueling-advantage critics learned from one-step transitions, and $\lambda$-return and soft Generalized Advantage Estimation (GAE) critics learned either (iii, iv) on-policy or (v, vi) from replayed trajectory segments. Each variant is evaluated across an entire random hyperparameter search on fourteen tasks, treating the hyperparameters as part of the algorithm. Learning soft advantages instead of soft Q-values does not improve performance, and the soft GAE critic adds nothing over the simpler $\lambda$-return critic. What does improve performance, most clearly on continuous control, is training a $\lambda$-return critic on replayed trajectory segments, even without importance weighting or trace cutting. This extends earlier evidence that uncorrected multi-step targets work well under replay to the full soft $\lambda$-return, entropy terms included. By design, the underlying ideas studied here extend beyond SAC, and our empirical findings aim to encourage their application to a broader class of reinforcement learning algorithms. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.05666 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-05T18:15:49.004Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).