Metraipolis › archive › dc_204275ead3af

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning — arXiv:2608.13596

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13596
License (as archived)
CC0-1.0
Archived
2026-10-19T02:21:00.216Z
SHA-256 of the archived text
70d5473e28cbd93d325996020963780bccd349e34049f924c0b6b598613eb44b
Stable URL of this page
https://metraipolis.com/documento/dc_204275ead3af

How to cite

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning — arXiv:2608.13596 — arXiv (en), https://arxiv.org/abs/2608.13596 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_204275ead3af (Metraipolis archive dc_204275ead3af, sha256 70d5473e28cbd93d325996020963780bccd349e34049f924c0b6b598613eb44b)
BibTeX
@misc{dc204275ead3af,
  title = {Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning — arXiv:2608.13596},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13596},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_204275ead3af; sha256 70d5473e28cbd93d325996020963780bccd349e34049f924c0b6b598613eb44b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13596. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales. We study an underexplored cross-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch. We ask whether useful capabilities can be transferred without explicit neuron-wise semantic alignment. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can already improve the recipient, we propose Activation-Prune-Merge (APM), an activation-guided framework for cross-scale fusion. APM constructs task-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55.5% to 60.6% over the original 3B recipient. RTE accuracy increases from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%. Analyses of injection ratios and sequential multi-stage fusion further suggest that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime. These results provide evidence that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.13596 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.850Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).