Metraipolis › archive › dc_500d6dc048f8

Training-Free Transformer Merging via Sequential Local Operator Alignment — arXiv:2610.06415

Source
arXiv
Original
https://arxiv.org/abs/2610.06415
License (as archived)
CC0-1.0
Archived
2026-10-06T05:31:38.086Z
SHA-256 of the archived text
b08e505fba2a074ec8b056ba5bc087e90b4ad6318a2ee49715b83e22579b609f
Stable URL of this page
https://metraipolis.com/documento/dc_500d6dc048f8

How to cite

Training-Free Transformer Merging via Sequential Local Operator Alignment — arXiv:2610.06415 — arXiv, https://arxiv.org/abs/2610.06415 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_500d6dc048f8 (Metraipolis archive dc_500d6dc048f8, sha256 b08e505fba2a074ec8b056ba5bc087e90b4ad6318a2ee49715b83e22579b609f)
BibTeX
@misc{dc500d6dc048f8,
  title = {Training-Free Transformer Merging via Sequential Local Operator Alignment — arXiv:2610.06415},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06415},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_500d6dc048f8; sha256 b08e505fba2a074ec8b056ba5bc087e90b4ad6318a2ee49715b83e22579b609f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06415. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Training-free model merging aims to combine multiple fine-tuned models into a single model without further optimization on labeled data. Yet, in transformers, independently merging individual layers can affect a shared attention computation because the query-key and value-output operators depend on composed matrices, overlooking the functional structure. Moreover, when merging earlier components, downstream components receive different activations than they do in the original model, thus, the merged and original execution paths no longer match. In this paper, we introduce Sequential Local Operator Alignment, a training-free method that merges transformers along the execution path of the partially merged model. Our method uses calibration data to estimate the local behavior of each functional component, aligns operators sequentially under the intermediate activation of the partially merged model, and subsequently factorizes the merged operators back into valid transformer parameters. We empirically show that this sequential step reduces error accumulation across layers. Furthermore, the proposed operator factorization step enables rank expansion, providing a principled mechanism for increasing multi-task capacity. We demonstrate that our approach generalizes across modalities, model scales, and varying numbers of tasks, from CLIP and RoBERTa to billion-parameter LLMs, and further extends naturally to the merging of LoRA-fine-tuned models. The results indicate improvements over strong merging baselines without requiring rank expansion, while optional expansion provides a further accuracy-inference-cost trade-off. Project link: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.06415 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.885Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).