Metraipolis › archive › dc_263f28feb175

A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic — arXiv:2610.07990

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.07990
License (as archived)
CC0-1.0
Archived
2026-10-07T11:23:45.577Z
SHA-256 of the archived text
1012ab25af8eee42050acb0b1ea2e31dad02b3ca8c0e8ab854d481701e304137
Stable URL of this page
https://metraipolis.com/documento/dc_263f28feb175

How to cite

A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic — arXiv:2610.07990 — arXiv (en), https://arxiv.org/abs/2610.07990 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_263f28feb175 (Metraipolis archive dc_263f28feb175, sha256 1012ab25af8eee42050acb0b1ea2e31dad02b3ca8c0e8ab854d481701e304137)
BibTeX
@misc{dc263f28feb175,
  title = {A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic — arXiv:2610.07990},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07990},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_263f28feb175; sha256 1012ab25af8eee42050acb0b1ea2e31dad02b3ca8c0e8ab854d481701e304137},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07990. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2610.07990 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:39.493Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).