Metraipolis › archive › dc_dc6be275c903
Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning — arXiv:2609.14327
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.14327
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T00:10:40.612Z
- SHA-256 of the archived text
7e95dec83701c7c9c96727acaa7a21b9e75a4739007c15110787e84b04a9c09c- Stable URL of this page
https://metraipolis.com/documento/dc_dc6be275c903
How to cite
Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning — arXiv:2609.14327 — arXiv (en), https://arxiv.org/abs/2609.14327 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc6be275c903 (Metraipolis archive dc_dc6be275c903, sha256 7e95dec83701c7c9c96727acaa7a21b9e75a4739007c15110787e84b04a9c09c)
BibTeX
@misc{dcdc6be275c903,
title = {Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning — arXiv:2609.14327},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.14327},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc6be275c903; sha256 7e95dec83701c7c9c96727acaa7a21b9e75a4739007c15110787e84b04a9c09c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14327. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Variance penalization is a principled approach to risk-sensitive reinforcement learning (RL) that explicitly trades expected return for policy stability. Existing methods require a dedicated second critic to estimate return variance online, adding architectural complexity and compounding estimation error during learning. We propose a nonparametric variance-penalized actor-critic (VPAC) framework that replaces the variance critic with statistically grounded online estimators based on bootstrapping and random scaling, techniques drawn from the statistical inference literature for stochastic approximation. These estimators require no auxiliary network, maintain a single-critic architecture, and produce variance penalties that are bounded by construction, enabling clean convergence analysis. We establish almost-sure convergence for both a variance-penalized Q-learning algorithm and a two-timescale actor-critic variant via the ordinary differential equation (ODE) method, requiring only that variance estimates remain bounded rather than consistent. Empirically, we evaluate across discrete and continuous stochastic environments, demonstrating that the proposed methods match or exceed the variance reduction achieved by the existing dual-critic VPAC baseline while eliminating the overhead of a second critic. We further validate on a high-temperature superconductor (HTS) manufacturing case study, where VPAC-RS (Random Scaling) achieves a 74% reduction in steady-state critical current variability and a 63% reduction in episode return standard deviation, translating directly to improved yield consistency. Our results establish nonparametric statistical inference as a practical and theoretically sound alternative to auxiliary critics for risk-sensitive RL. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.LG). Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2609.14327 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T23:54:14.690Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).