Metraipolis › archive › dc_1f4e754147ad

MonoMoE: An Efficient Fused Mega-kernel for Quantized MoE Decoding — arXiv:2609.04244

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04244
License (as archived)
CC0-1.0
Archived
2026-10-16T22:42:12.848Z
SHA-256 of the archived text
6febfff4f202e35983f2a13c8b955d0ee2fa6a4247c633c3375d0786daacaecd
Stable URL of this page
https://metraipolis.com/documento/dc_1f4e754147ad

How to cite

MonoMoE: An Efficient Fused Mega-kernel for Quantized MoE Decoding — arXiv:2609.04244 — arXiv (en), https://arxiv.org/abs/2609.04244 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1f4e754147ad (Metraipolis archive dc_1f4e754147ad, sha256 6febfff4f202e35983f2a13c8b955d0ee2fa6a4247c633c3375d0786daacaecd)
BibTeX
@misc{dc1f4e754147ad,
  title = {MonoMoE: An Efficient Fused Mega-kernel for Quantized MoE Decoding — arXiv:2609.04244},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04244},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1f4e754147ad; sha256 6febfff4f202e35983f2a13c8b955d0ee2fa6a4247c633c3375d0786daacaecd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04244. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) layers increase model capacity without proportionally increasing arithmetic, but their sparse expert computation is difficult to execute efficiently during autoregressive decode. Existing grouped and batched GEMMs are token-major: they construct expert-local token tiles and obtain parallelism from the token dimension. When few tokens reach each expert, this organization incurs tile padding and preprocessing, launches short-lived grids that underutilize memory bandwidth, and exposes quantization, activation, and reduction as separate stages. We present \textbf{MonoMoE}, a weight-major persistent megakernel for block-wise quantized MoE decode. MonoMoE places the complete decode-step token tile on the fine-grained tensor-core $N$ dimension and partitions CTAs over expert-weight tiles, eliminating expert-local token materialization and reducing padded arithmetic. A persistent grid fuses routing, top-$k$ selection, quantization, both expert projections, activation, and reduction in one launch; warp specialization and readiness flags overlap auxiliary work with the dominant expert-weight stream. MonoMoE is integrated with vLLM and supports multiple model shapes through generated kernel specializations and offline schedule tuning. On NVIDIA H200 GPUs, MonoMoE accelerates the complete routed-MoE operator by up to $\mathbf{1.54\times}$ over vLLM Triton Grouped GEMM, is $\mathbf{2.20}$--$\mathbf{3.84\times}$ faster than FlashMoE-FP8 adaptation across various models and batch sizes, and reduces end-to-end time per output token by up to $\mathbf{18.7\%}$ across the evaluated FP8 models, while preserving task accuracy. The MonoMoE implementation and supporting artifacts are open source and available in the \href{ this https URL }{FlashInfer repository}.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AR).
Categorie arXiv: Hardware Architecture (cs.AR).
Pagina del lavoro: https://arxiv.org/abs/2609.04244 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:26:48.903Z dalla pagina https://arxiv.org/catchup/cs.AR/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).