Metraipolis › archive › dc_59afe7de1a2d
Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection — arXiv:2608.12912
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.12912
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T04:27:28.414Z
- SHA-256 of the archived text
e74da8d8d9a65f6b9931570642e89ff3b14b4fda513c8412a7b7de5e88827c17- Stable URL of this page
https://metraipolis.com/documento/dc_59afe7de1a2d
How to cite
Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection — arXiv:2608.12912 — arXiv (en), https://arxiv.org/abs/2608.12912 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_59afe7de1a2d (Metraipolis archive dc_59afe7de1a2d, sha256 e74da8d8d9a65f6b9931570642e89ff3b14b4fda513c8412a7b7de5e88827c17)
BibTeX
@misc{dc59afe7de1a2d,
title = {Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection — arXiv:2608.12912},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.12912},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_59afe7de1a2d; sha256 e74da8d8d9a65f6b9931570642e89ff3b14b4fda513c8412a7b7de5e88827c17},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12912. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
This paper considers the overestimation bias problem of Q-learning in the setting of a large action space, for the purpose of relieving the bottleneck of existing methods. We find that the large action space increases the randomness in Q-value estimation. The randomness makes two paradigms that drive the major literature on the overestimation problem have their own bottlenecks: the coupling paradigm, i.e., the optimal action and its Q-value are estimated with the same Q-function, always has a positive bias. This is because randomness leads to some actions having abnormally high estimated values than their true values, and the coupling methods prefer these actions. The decoupling paradigm, i.e., the optimal action and its Q-value are estimated with two independent Q-functions, always has a negative bias. This is because randomness increases the estimation gap between the two independent Q-tables for the same action. This paper shows that action intersection can be a simple yet powerful strategy to relieve these bottlenecks. The action intersection strategy enables semi-decoupling via two designs: (1) it allows two Q-functions to share a certain fraction of trajectory data; (2) if a data sample is shared, each Q-function is updated using the coupling paradigm; otherwise, using the decoupling paradigm. Two properties make the action intersection strategy powerful: (1) attaining a large bias range, i.e., varying the data sharing fraction, the estimation bias varies from underestimating to overestimating; (2) fine granularity: the action intersection size can be made arbitrarily finer to enable finer control. We consider two experiment settings, i.e., tabular and deep RL, deep RL experiments show that our method outperforms several SOTA baselines drastically; tabular experiments reveal why our method can achieve superior performance. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.12912 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T04:20:35.832Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).