Metraipolis › archive › dc_36eeeb7733b5
Rethinking Self-Distillation for Multi-Teacher Capability Merging — arXiv:2610.04272
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.04272
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:26:05.898Z
- SHA-256 of the archived text
13ab8a9a066edbd36ba58ad1ae3876a02eb75f9f7d5a79b3f9f859611a66bd21- Stable URL of this page
https://metraipolis.com/documento/dc_36eeeb7733b5
How to cite
Rethinking Self-Distillation for Multi-Teacher Capability Merging — arXiv:2610.04272 — arXiv, https://arxiv.org/abs/2610.04272 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_36eeeb7733b5 (Metraipolis archive dc_36eeeb7733b5, sha256 13ab8a9a066edbd36ba58ad1ae3876a02eb75f9f7d5a79b3f9f859611a66bd21)
BibTeX
@misc{dc36eeeb7733b5,
title = {Rethinking Self-Distillation for Multi-Teacher Capability Merging — arXiv:2610.04272},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.04272},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_36eeeb7733b5; sha256 13ab8a9a066edbd36ba58ad1ae3876a02eb75f9f7d5a79b3f9f859611a66bd21},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04272. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Combining capabilities of multiple expert models trained starting from the same base checkpoint has become increasingly common in frontier language-model post-training. Recent trends suggest that multi-teacher on-policy distillation (MOPD) outperforms conventional off-policy methods. However, despite the higher inference and environment interaction costs incurred by MOPD, we find that much of its reported accuracy gain is due to certain training design choices and hyperparameter optimization, rather than the algorithm itself. We conduct a controlled self-distillation study across two multi-teacher settings, four models, and eleven benchmarks, comparing off-policy methods, namely supervised fine-tuning (SFT) and soft-label distillation, to hybrid teacher-prefix distillation and MOPD. We found that all four methods achieve \textit{nearly identical} accuracy. However, MOPD uses $14.8$--$23.1\times$ SFT's training GPU-hours. We also revisit four recently published comparisons between on-policy and off-policy distillation and find that the reported on-policy gains shrink substantially when SFT baselines are trained on rejection-sampled teacher trajectories and use independently tuned hyperparameters. As a training-free alternative, we also find that simple weight-merging methods can recover expert capabilities with minutes of CPU merging time, although their accuracy degrades as model size decreases and task interference increases. Overall, our results question recent gains reported due to MOPD and suggest careful tuning of more efficient off-policy baselines as a viable alternative.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.04272 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.823Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).