Metraipolis › archive › dc_8059fc51ebfa

CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training — arXiv:2610.05744

Source: arXiv
Original: https://arxiv.org/abs/2610.05744
License (as archived): CC0-1.0
Archived: 2026-10-06T05:27:08.786Z
SHA-256 of the archived text: 4e1da187b23abdbe58826f02528965c661fa4cd847fe907ebba9acbca0284dcf
Stable URL of this page: https://metraipolis.com/documento/dc_8059fc51ebfa

How to cite

CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training — arXiv:2610.05744 — arXiv, https://arxiv.org/abs/2610.05744 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8059fc51ebfa (Metraipolis archive dc_8059fc51ebfa, sha256 4e1da187b23abdbe58826f02528965c661fa4cd847fe907ebba9acbca0284dcf)

BibTeX

@misc{dc8059fc51ebfa,
  title = {CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training — arXiv:2610.05744},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05744},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8059fc51ebfa; sha256 4e1da187b23abdbe58826f02528965c661fa4cd847fe907ebba9acbca0284dcf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05744. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) has been widely adopted in recent large language model (LLM) architectures. However, scaling up MoE in LLM training introduces system-level challenges on training, where non-uniform token routing can lead to highly imbalanced workloads across experts and devices, further destabilizing the training process. With trillion-scale LLMs, imbalanced expert workloads further amplify the resource cost of MoE training, resulting in degraded training efficiency and hardware utilization for underloaded experts, while hot experts require additional resources to accommodate excessive workloads. Recent studies address imbalanced MoE training through intricate parallelism strategies or resource reallocation. However, these system-level approaches often introduce additional resource requirements and considerable orchestration complexity, which become increasingly difficult to afford when training trillion-parameter LLMs under constrained computational resources. This work introduces CIPHER-MoE, which mitigates MoE workload imbalance while keeping the router's token-side Top-K selection unchanged. CIPHER-MoE applies affinity-aware Expert-to-Token filtering with explicit capacity control to reduce hotspot expert workloads without additional hardware resources or complex runtime design. The proposed method has been evaluated on large-scale MoE models, including DeepSeek-V4-Pro, showing up to 64.9 percentage points Top-1 expert workload reduction and 1.10$\times$-1.94$\times$ training acceleration, while preserving the training quality. The source code will be released soon.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.05744 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:06.067Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).