Metraipolis › archive › dc_74a7ff281166
GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts — arXiv:2611.04222
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04222
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T12:29:45.651Z
- SHA-256 of the archived text
69357abf573baf87c0971668313c7c7da9d5906a80450900fb01ad2cd22701ec- Stable URL of this page
https://metraipolis.com/documento/dc_74a7ff281166
How to cite
GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts — arXiv:2611.04222 — arXiv (en), https://arxiv.org/abs/2611.04222 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_74a7ff281166 (Metraipolis archive dc_74a7ff281166, sha256 69357abf573baf87c0971668313c7c7da9d5906a80450900fb01ad2cd22701ec)
BibTeX
@misc{dc74a7ff281166,
title = {GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts — arXiv:2611.04222},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04222},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_74a7ff281166; sha256 69357abf573baf87c0971668313c7c7da9d5906a80450900fb01ad2cd22701ec},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04222. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Under expert parallelism, mixture-of-experts (MoE) layers pay all-to-all token dispatch, grouped GEMMs, and a combine step in every layer. In decode each rank holds only 1-256 tokens: dispatch latency and weight traffic dominate while tensor-core utilization is inherently low. In fused operators such as DeepGEMM's Mega MoE, dispatch and computation overlap only by sharing every SM: dispatch warps compete with TMA/MMA and epilogue warps for registers, shared memory, and issue slots; every phase shares one CTA resource budget, synchronization spans the whole SM array, and every replay ends with a workspace-clean pass and a cross-rank barrier. We present GC-MoE, an open-source FP8/FP4 MoE inference operator for NVIDIA Hopper and Blackwell GPUs that uses CUDA Green Contexts to split the device into a small communication partition and a large compute partition. Each role gets its own kernel, resource budget, and synchronization domain, so dispatch and GEMM tune independently. A three-node CUDA Graph (four on Hopper with fused shared experts) launches the dispatch and grouped-GEMM kernels as concurrent roots; a token-ring handshake starts computation on the first filled ring blocks, and double-buffered receive counters remove the trailing cross-rank barrier between replays. The compute partition runs operand-swapped tcgen05 block-scaled MMA on Blackwell and register-A WGMMA with in-register FP4 decoding on Hopper, with fused SwiGLU, requantization, and optional shared experts. On 8 H20 GPUs GC-MoE accelerates DeepSeek-V4.1-Flash decode by 1.188x on average on both the routed and shared-expert suites (peaks 1.296x and 1.353x); on 8 B300 GPUs it accelerates GLM-5.2 and DeepSeek-V4.1-Flash by 1.106x (1.123x with fused shared experts, peak 1.209x), beating Mega MoE at all 54 measured points at matched accuracy. GC-MoE reserves only 20 of 148 SMs on B300 for dispatch. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR); Machine Learning (cs.LG); Performance (cs.PF). Pagina del lavoro: https://arxiv.org/abs/2611.04222 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T12:26:47.255Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).