Metraipolis › archive › dc_a5b9811efa89

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs — arXiv:2609.25655

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.25655
License (as archived)
CC0-1.0
Archived
2026-10-08T13:46:34.218Z
SHA-256 of the archived text
a7c55e64fab928e423474ec6ecb8cbbfaa8108aa99ecb6fbef24d74b149c2ade
Stable URL of this page
https://metraipolis.com/documento/dc_a5b9811efa89

How to cite

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs — arXiv:2609.25655 — arXiv (en), https://arxiv.org/abs/2609.25655 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a5b9811efa89 (Metraipolis archive dc_a5b9811efa89, sha256 a7c55e64fab928e423474ec6ecb8cbbfaa8108aa99ecb6fbef24d74b149c2ade)
BibTeX
@misc{dca5b9811efa89,
  title = {From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs — arXiv:2609.25655},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25655},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a5b9811efa89; sha256 a7c55e64fab928e423474ec6ecb8cbbfaa8108aa99ecb6fbef24d74b149c2ade},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25655. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectures such as Mixture-of-Experts (MoE) models. This shift raises a key question for parameter-efficient fine-tuning (PEFT): at what granularity should parameters be selected and updated? Existing PEFT methods such as LoRA operate on predefined weight matrices, while expert-level sparse tuning methods update entire selected experts. However, we observe that activated experts are internally sparse, with only a small fraction of intermediate channels strongly responding to downstream tasks, indicating that expert-level adaptation is still too coarse. We propose NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts. NSFT decomposes each expert along the intermediate dimension into structured channel groups and selects task-relevant sub-experts by combining routing importance with intra-expert activation saliency. To optimize sparse partial updates, NSFT further introduces learning-rate scaling and dynamic gradient scaling to compensate for the reduced effective update magnitude. Experiments on OLMoE and Ling-mini-2.0 across challenging domain-specific tasks and general benchmarks show that NSFT consistently outperforms representative PEFT and expert-level sparse tuning baselines, while using substantially fewer trainable parameters and preserving competitive general capability. These results suggest that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.25655 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:18:38.250Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).