Metraipolis › archive › dc_84aa35754d18
Convex-Concave Reinforcement Learning — arXiv:2610.09108
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09108
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:10:33.944Z
- SHA-256 of the archived text
c0390aa029c62c1c2e679dec3302f4bd3d44de1d9e39117495ce1d9abbbb5a19- Stable URL of this page
https://metraipolis.com/documento/dc_84aa35754d18
How to cite
Convex-Concave Reinforcement Learning — arXiv:2610.09108 — arXiv (en), https://arxiv.org/abs/2610.09108 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84aa35754d18 (Metraipolis archive dc_84aa35754d18, sha256 c0390aa029c62c1c2e679dec3302f4bd3d44de1d9e39117495ce1d9abbbb5a19)
BibTeX
@misc{dc84aa35754d18,
title = {Convex-Concave Reinforcement Learning — arXiv:2610.09108},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09108},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84aa35754d18; sha256 c0390aa029c62c1c2e679dec3302f4bd3d44de1d9e39117495ce1d9abbbb5a19},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09108. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Policy learning drives many of the most consequential and heavily-invested applications of reinforcement learning today. Yet the core optimization problem it rests on (maximizing expected return) is notoriously non-convex, even under a direct policy parameterization, and the field has largely responded by avoiding it: optimizing convex surrogate approximations of the return under trust-region constraints (NPG, TRPO, PPO, AWR). We show that this seemingly unstructured problem is not actually structureless. In log-density-ratio coordinates $y := \log[\pi/\pi_n]$, the exact per-iteration objective, computable via per-decision importance sampling (PDIS), is a difference-of-convex-constrained difference-of-convex (DC-constrained DC) program. This structure lets us move beyond surrogate approximations: it recovers CPI, NPG, TRPO, and AWR as special cases along interpretable axes, and it opens a multi-step axis $k$ that couples consecutive decisions. We solve the per-iteration program with sequential convex programming (SCP), the standard solver for difference-of-convex problems, and give convergence guarantees under mild conditions, bridging the difference-of-convex optimization and RL literatures. Empirically, multi-step Convex-Concave RL (CCRL) wins on diagnostic MDPs where credit must propagate across a horizon (its advantage growing with the dependency length), is competitive with a tuned PPO on classic control, and on a realistic, stochastic, mid-horizon healthcare domain converges markedly faster than tuned PPO to the same near-optimal survival, with an 11.3% higher area under the training curve. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.09108 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:14:06.049Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).