Metraipolis › archive › dc_81cf267d84fe

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training — arXiv:2610.34272

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.34272
License (as archived)
CC0-1.0
Archived
2026-10-27T08:01:13.540Z
SHA-256 of the archived text
d86f59d56949e44f9c40c450b2c7755c22269606ec480f9ecc8951ed5db132dd
Stable URL of this page
https://metraipolis.com/documento/dc_81cf267d84fe

How to cite

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training — arXiv:2610.34272 — arXiv (en), https://arxiv.org/abs/2610.34272 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_81cf267d84fe (Metraipolis archive dc_81cf267d84fe, sha256 d86f59d56949e44f9c40c450b2c7755c22269606ec480f9ecc8951ed5db132dd)
BibTeX
@misc{dc81cf267d84fe,
  title = {DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training — arXiv:2610.34272},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.34272},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_81cf267d84fe; sha256 d86f59d56949e44f9c40c450b2c7755c22269606ec480f9ecc8951ed5db132dd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34272. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$-$5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$-$90.1\%$, while maintaining comparable multi-domain performance.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.34272 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:30.888Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).