Metraipolis › archive › dc_f732dcbb806c
Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning — arXiv:2609.03150
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.03150
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:41:04.416Z
- SHA-256 of the archived text
e60db4a84cb913943b982ccdb2dd91ca301e9985e13c0ff44e97adb120009971- Stable URL of this page
https://metraipolis.com/documento/dc_f732dcbb806c
How to cite
Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning — arXiv:2609.03150 — arXiv (en), https://arxiv.org/abs/2609.03150 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f732dcbb806c (Metraipolis archive dc_f732dcbb806c, sha256 e60db4a84cb913943b982ccdb2dd91ca301e9985e13c0ff44e97adb120009971)
BibTeX
@misc{dcf732dcbb806c,
title = {Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning — arXiv:2609.03150},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.03150},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f732dcbb806c; sha256 e60db4a84cb913943b982ccdb2dd91ca301e9985e13c0ff44e97adb120009971},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03150. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multi-domain fine-tuning often combines MoE routing with LoRA, assuming that token-level routing separates domain-specific updates. We test this assumption in MoE+LoRA using Python code paired with biomedical text and mathematical reasoning. Although these domains show near-disjoint expert routing, adding biomedical data substantially increases code perplexity, indicating that routing separation alone may not prevent negative transfer. To localize the failure, we introduce Jaccard routing overlap and adapter-gradient cosine similarity, which measure expert sharing and update compatibility, respectively. These diagnostics indicate that interference arises mostly from nearly orthogonal domain gradients competing within the same low-rank adapter subspace. We address this issue with SpawnLoRA, which dynamically adds gated sub-adapters inside MoE experts when adapter-level contention is detected, while keeping the router fixed. We evaluate SpawnLoRA on Phi-tiny-MoE-instruct and OLMoE-1B-7B across multiple mixture settings and find that it effectively reduces negative transfer compared with standard and rank-adaptive LoRA. These results demonstrate that structural separation inside experts provides benefits beyond routing or rank expansion alone. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.03150 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T01:11:43.584Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).