Metraipolis › archive › dc_df609561b59d
Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs — arXiv:2611.04376
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04376
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T21:40:01.154Z
- SHA-256 of the archived text
2c733883c936c29ff3308563af22a875e43524c5f90a8fcb1ebfc40678eda8de- Stable URL of this page
https://metraipolis.com/documento/dc_df609561b59d
How to cite
Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs — arXiv:2611.04376 — arXiv (en), https://arxiv.org/abs/2611.04376 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_df609561b59d (Metraipolis archive dc_df609561b59d, sha256 2c733883c936c29ff3308563af22a875e43524c5f90a8fcb1ebfc40678eda8de)
BibTeX
@misc{dcdf609561b59d,
title = {Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs — arXiv:2611.04376},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04376},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_df609561b59d; sha256 2c733883c936c29ff3308563af22a875e43524c5f90a8fcb1ebfc40678eda8de},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04376. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoeResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoeResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.04376 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T21:35:49.667Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).