Metraipolis › archive › dc_bc7d1aa70f5d

CASS: Contribution-Aware Structured Sparsity for Model Merging — arXiv:2609.34184

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34184
License (as archived)
CC0-1.0
Archived
2026-10-07T20:57:31.365Z
SHA-256 of the archived text
823201d97e47e43905b949352cab6e58d65474bb8e68c969c374706bed62db9b
Stable URL of this page
https://metraipolis.com/documento/dc_bc7d1aa70f5d

How to cite

CASS: Contribution-Aware Structured Sparsity for Model Merging — arXiv:2609.34184 — arXiv (en), https://arxiv.org/abs/2609.34184 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bc7d1aa70f5d (Metraipolis archive dc_bc7d1aa70f5d, sha256 823201d97e47e43905b949352cab6e58d65474bb8e68c969c374706bed62db9b)
BibTeX
@misc{dcbc7d1aa70f5d,
  title = {CASS: Contribution-Aware Structured Sparsity for Model Merging — arXiv:2609.34184},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34184},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bc7d1aa70f5d; sha256 823201d97e47e43905b949352cab6e58d65474bb8e68c969c374706bed62db9b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34184. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose \textbf{C}ontribution-\textbf{A}ware \textbf{S}tructured \textbf{S}parsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering as an effective way to reduce merging interference. Extensive experiments across vision (ViT, 20 tasks) and language (RoBERTa, 8 tasks; Qwen2.5, 4 tasks) benchmarks demonstrate that CASS improves a range of representative merging baselines.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.34184 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.473Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).