Metraipolis › archive › dc_1bfefdd0de0e

MeRoTune: RoPE-Safe Merging with a Tunable Dial — arXiv:2609.07971

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.07971
License (as archived)
CC0-1.0
Archived
2026-10-16T12:48:33.708Z
SHA-256 of the archived text
d19ee3c9b08c58c7c47f8ada477488ad60efddca39f51bc5aad93bee4313e347
Stable URL of this page
https://metraipolis.com/documento/dc_1bfefdd0de0e

How to cite

MeRoTune: RoPE-Safe Merging with a Tunable Dial — arXiv:2609.07971 — arXiv (en), https://arxiv.org/abs/2609.07971 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1bfefdd0de0e (Metraipolis archive dc_1bfefdd0de0e, sha256 d19ee3c9b08c58c7c47f8ada477488ad60efddca39f51bc5aad93bee4313e347)
BibTeX
@misc{dc1bfefdd0de0e,
  title = {MeRoTune: RoPE-Safe Merging with a Tunable Dial — arXiv:2609.07971},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.07971},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1bfefdd0de0e; sha256 d19ee3c9b08c58c7c47f8ada477488ad60efddca39f51bc5aad93bee4313e347},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07971. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When you merge two fine-tuned models from the same base checkpoint by simply averaging their weights, you implicitly assume their attention subspaces are still aligned. Recent work attempts to fix misalignments by learning an invertible correction matrix, $M$, for each model's query and key projections. This correction cancels out---using $M$ on the query side and $M^{-T}$ on the key side---right before the dot product. However, this cancellation is only exact if nothing sits between the projection and the dot product. In reality, almost all modern open-weight language models put a rotary position embedding (RoPE) exactly there. In this paper, we show that this cancellation is exact under RoPE if and only if $M$ commutes with RoPE's per-position rotation. We derive the specific class of matrices where this holds: a scaled rotation acting independently within each RoPE frequency pair. This forms a strict, low-dimensional subset of the unconstrained matrices that current methods normally train. Building on this, we turn this constrained matrix class into a new merging method. While keeping the base weights entirely frozen, two fine-tunes each learn their own RoPE-compliant correction matrices. We optimize these corrections against a chosen blend ratio so the final result can be adjusted post-hoc like a dial, rather than locked into a single fixed merge. Our default approach trains at one fixed blend ratio, similar to how LoRA sets its scaling hyperparameter in advance. We also experiment with resampling the blend ratio randomly at every training step, and we report the results of both approaches.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.07971 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.153Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).