Metraipolis › archive › dc_ba5fec9ff3b4

GrowMTP: Can RL Grow Its Own Draft Head? — arXiv:2609.16648

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.16648
License (as archived)
CC0-1.0
Archived
2026-10-15T19:23:27.115Z
SHA-256 of the archived text
755d4b5a02ef13b0d7d18f4944b21bde93cd0bb3a4d122741f2f8894f951065c
Stable URL of this page
https://metraipolis.com/documento/dc_ba5fec9ff3b4

How to cite

GrowMTP: Can RL Grow Its Own Draft Head? — arXiv:2609.16648 — arXiv (en), https://arxiv.org/abs/2609.16648 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ba5fec9ff3b4 (Metraipolis archive dc_ba5fec9ff3b4, sha256 755d4b5a02ef13b0d7d18f4944b21bde93cd0bb3a4d122741f2f8894f951065c)
BibTeX
@misc{dcba5fec9ff3b4,
  title = {GrowMTP: Can RL Grow Its Own Draft Head? — arXiv:2609.16648},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.16648},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ba5fec9ff3b4; sha256 755d4b5a02ef13b0d7d18f4944b21bde93cd0bb3a4d122741f2f8894f951065c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16648. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) post-training drives the frontier capabilities of large language models, with its wall-clock dominated by autoregressive rollout generation. Speculative decoding is an established remedy for this bottleneck, but existing draft heads must be pretrained or warmed up before RL, introducing substantial training cost outside the RL run to be accelerated. We observe that RL training itself provides both conditions required for online draft-head training: its rollout distribution is far narrower than that of pretraining, and its verification step continuously produces supervision signals aligned with this distribution. Building on these observations, we propose GrowMTP, which uses this supervision to train a draft head from scratch entirely within the RL loop, with all head updates detached from the policy backbone. On Qwen3-4B (no draft head), MiMo-7B-SFT (weak head), and Qwen3.5-4B-Base (strong head), GrowMTP achieves rollout speedups of 2.13x, 1.93x, and 1.36x, and end-to-end speedups of 1.60x, 1.41x, and 1.20x, respectively. GrowMTP therefore serves existing RL training frameworks as a modular component, particularly offering a from-scratch acceleration path for models without pretrained draft heads.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.16648 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:32:32.903Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).