Metraipolis › archive › dc_ef920fb8c01a

Model Merging Does Not Require Uniform LoRA Ranks — arXiv:2611.06486

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.06486
License (as archived): CC0-1.0
Archived: 2026-11-05T18:17:59.330Z
SHA-256 of the archived text: 47bf29aa743372410aef2c63269e8b17561636707151f4ab1a20a0b5949a114a
Stable URL of this page: https://metraipolis.com/documento/dc_ef920fb8c01a

How to cite

Model Merging Does Not Require Uniform LoRA Ranks — arXiv:2611.06486 — arXiv (en), https://arxiv.org/abs/2611.06486 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef920fb8c01a (Metraipolis archive dc_ef920fb8c01a, sha256 47bf29aa743372410aef2c63269e8b17561636707151f4ab1a20a0b5949a114a)

BibTeX

@misc{dcef920fb8c01a,
  title = {Model Merging Does Not Require Uniform LoRA Ranks — arXiv:2611.06486},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06486},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef920fb8c01a; sha256 47bf29aa743372410aef2c63269e8b17561636707151f4ab1a20a0b5949a114a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06486. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following, multilingual) at four ranks and three model scales, then merge with six methods at uniform ranks and at each domain's own best rank. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp. The split has a simple cause. Merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp. Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8$\times$ rank range (0.7 pp, against 6.7 pp for TIES at 4B scale). Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.06486 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:12:49.059Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).