Metraipolis › archive › dc_65a104aa83ac

Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning — arXiv:2610.21682

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.21682
License (as archived)
CC0-1.0
Archived
2026-10-16T09:28:30.274Z
SHA-256 of the archived text
2cd6f92e4a8686d6a998f89f7f484e26d0d4280150f9da6f6e85411f6ba6c6b1
Stable URL of this page
https://metraipolis.com/documento/dc_65a104aa83ac

How to cite

Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning — arXiv:2610.21682 — arXiv (en), https://arxiv.org/abs/2610.21682 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_65a104aa83ac (Metraipolis archive dc_65a104aa83ac, sha256 2cd6f92e4a8686d6a998f89f7f484e26d0d4280150f9da6f6e85411f6ba6c6b1)
BibTeX
@misc{dc65a104aa83ac,
  title = {Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning — arXiv:2610.21682},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21682},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_65a104aa83ac; sha256 2cd6f92e4a8686d6a998f89f7f484e26d0d4280150f9da6f6e85411f6ba6c6b1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21682. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous this http URL resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout--training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.21682 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:17.982Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).