Metraipolis › archive › dc_9db6ef21185b
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation — arXiv:2608.03092
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03092
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:18:32.439Z
- SHA-256 of the archived text
73991dd1023cbb0419fa508e45756b2831d40f3f5dae8010e3513cefdc763a25- Stable URL of this page
https://metraipolis.com/documento/dc_9db6ef21185b
How to cite
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation — arXiv:2608.03092 — arXiv (en), https://arxiv.org/abs/2608.03092 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9db6ef21185b (Metraipolis archive dc_9db6ef21185b, sha256 73991dd1023cbb0419fa508e45756b2831d40f3f5dae8010e3513cefdc763a25)
BibTeX
@misc{dc9db6ef21185b,
title = {SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation — arXiv:2608.03092},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03092},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9db6ef21185b; sha256 73991dd1023cbb0419fa508e45756b2831d40f3f5dae8010e3513cefdc763a25},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03092. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We aim to improve model performance in multi-reward reinforcement learning training process. Existing Group reward-Decoupled Normalization Policy Optimization (GDPO) has mitigated the issue of reward signals masking one another during direct scalarization by normalizing each reward dimension separately before aggregation. However, our experiments show that GDPO still struggles to balance reward signals with different granularities. Specifically, in some particular training tasks, the model may receive a dense reward that assigns fine-grained scores ranging from 0.1 to 1.0, together with a sparse reward that provides only binary feedback of either 0 or 1. In such cases, we find that the sparse reward may provide an insufficient optimization signal, preventing its corresponding capability from being effectively reinforced. Therefore, how can we strengthen the optimization signal from the sparse reward without sacrificing the capability already learned from the fine-grained reward? To overcome this limitation, we propose Specialize-and-Merge Online Policy Distillation (SMOPD), a two-stage training method for multi-reward optimization. Stage1-Specialize: SMOPD first employs reward-priority configurations to train multiple reward-specialized teachers, allowing each reward to be learned under conditions where its signal can effectively drive optimization. Stage2-Merge: SMOPD then utilizes online policy distillation to combine the reward-specialized capabilities of these teachers into a single student policy, while maintaining balanced task-level optimization. To validate our method, we conduct experiments on two multi-reward settings: complementary rewards(tool-calling accuracy and format) and conflicting rewards (helpful and harmless rewards). Based on above settings, SMOPD outperforms GDPO across 1.5B, 3B and 7B backbones. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.03092 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:05:52.994Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).