Metraipolis › archive › dc_9f29f5df8a67

DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignment — arXiv:2609.27572

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.27572
License (as archived): CC0-1.0
Archived: 2026-10-08T09:12:48.210Z
SHA-256 of the archived text: 7d01aad9f32a526b699c094f0cd49c8ef4399494ccc1e969166d0abd9d1bf9b3
Stable URL of this page: https://metraipolis.com/documento/dc_9f29f5df8a67

How to cite

DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignment — arXiv:2609.27572 — arXiv (en), https://arxiv.org/abs/2609.27572 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9f29f5df8a67 (Metraipolis archive dc_9f29f5df8a67, sha256 7d01aad9f32a526b699c094f0cd49c8ef4399494ccc1e969166d0abd9d1bf9b3)

BibTeX

@misc{dc9f29f5df8a67,
  title = {DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignment — arXiv:2609.27572},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27572},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9f29f5df8a67; sha256 7d01aad9f32a526b699c094f0cd49c8ef4399494ccc1e969166d0abd9d1bf9b3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27572. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) has emerged as a key paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing reward systems, such as rule-based and reward-model-based, often exhibit issues such as unstable optimization and reward hacking. In this work, we revisit the general reasoning of LLMs from a geometric perspective, conceptualizing it as a coupled manifold composed of three interdependent sub-manifolds: logical deduction, evaluation, and representation. Based on this perspective, response generation in RL can be interpreted as a decoupling process from the evaluation manifold, while reward estimation corresponds to a decoupling process from the logical deduction manifold. The limitations of rule-based and reward-model RL systems can be geometrically interpreted as the mismatch of policy-reward manifolds during RL process. To address the aforementioned misalignment, we propose Decoupling and Coupling Reinforcement Learning (DCRL) framework, which incorporates two key components: (1) a syllogistic logic-based prompt evolution mechanism that dynamically refines reward rubrics to enhance the expressiveness of the reward manifold; and (2) a policy-reward re-coupling mechanism that jointly updates the reward and policy models, ensuring consistent evaluation and mitigating manifold mismatch during training. Theoretical analysis and extensive experiments across multiple reasoning domains demonstrate that DCRL consistently outperforms both rule-based and reward-model baselines. Notably, a Qwen3-4B model trained under DCRL surpasses a Qwen3-32B baseline and approaches the performance of a Qwen3-235B model, highlighting superior effectiveness and generalization in RL.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.27572 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T08:13:26.692Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).