Metraipolis › archive › dc_641d8fa0257c
Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging — arXiv:2609.32303
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32303
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:55:17.156Z
- SHA-256 of the archived text
85253276c010c91638a5eae19bb0d5986bcb6058162e567cd004cc65dbb45fd2- Stable URL of this page
https://metraipolis.com/documento/dc_641d8fa0257c
How to cite
Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging — arXiv:2609.32303 — arXiv (en), https://arxiv.org/abs/2609.32303 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_641d8fa0257c (Metraipolis archive dc_641d8fa0257c, sha256 85253276c010c91638a5eae19bb0d5986bcb6058162e567cd004cc65dbb45fd2)
BibTeX
@misc{dc641d8fa0257c,
title = {Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging — arXiv:2609.32303},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32303},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_641d8fa0257c; sha256 85253276c010c91638a5eae19bb0d5986bcb6058162e567cd004cc65dbb45fd2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32303. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Domain experts trained from a shared checkpoint can be merged into one model through on-policy distillation (OPD), where they act as teachers supervising a student on its own trajectories. One upstream choice is rarely examined: whether to build each expert with supervised fine-tuning (SFT) or reinforcement learning (RL). Yet equally strong teachers need not be equally good teachers. We probe this choice through controlled single-teacher OPD, a building block of multi-teacher OPD: in Agentic, Reasoning, and Perception, comparably performing SFT and RL teachers are trained from Qwen3.5-9B, each guiding a student initialized from it. At their best checkpoints, RL-guided students outperform SFT-guided students by 4.27, 1.50, and 0.86 percentage points in Agentic, Reasoning, and Perception, respectively, and recover more of their teachers' performance gains over the base model. The contrast is clearest in Agentic, where the best SFT-guided student recovers only 44.44% of its teacher's gain, whereas the best RL-guided student recovers 115.00%, surpassing its teacher. Our analysis points to an explanation: RL teachers stay much closer to the shared initialization in parameter space than SFT teachers and are therefore easier for their students to follow. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.32303 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:00.748Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).