Metraipolis › archive › dc_4ecc783af28a

MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference — arXiv:2611.01704

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.01704
License (as archived): CC0-1.0
Archived: 2026-11-03T11:06:41.437Z
SHA-256 of the archived text: c1d64164a1d501fbc10bfd324e1d1fa86fbb786e21b2b9d8e288876c2eae2697
Stable URL of this page: https://metraipolis.com/documento/dc_4ecc783af28a

How to cite

MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference — arXiv:2611.01704 — arXiv (en), https://arxiv.org/abs/2611.01704 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4ecc783af28a (Metraipolis archive dc_4ecc783af28a, sha256 c1d64164a1d501fbc10bfd324e1d1fa86fbb786e21b2b9d8e288876c2eae2697)

BibTeX

@misc{dc4ecc783af28a,
  title = {MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference — arXiv:2611.01704},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01704},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4ecc783af28a; sha256 c1d64164a1d501fbc10bfd324e1d1fa86fbb786e21b2b9d8e288876c2eae2697},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01704. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) architecture has been widely adopted in large language models (LLMs) since it reduces the computational cost through sparse expert activation. However, the massive parameter scale of MoE models hinders their deployment on resource-constrained edge devices. Although existing mixed-precision quantization reduces the model size, they are inherently static, fixing the bit-width of each expert across decoding steps. Such static schemes yield suboptimal performance as they overlook the dynamic variation in expert importance during decoding. Furthermore, they fail to adapt to the dynamic latency constraint driven by the fluctuating system utilization. To address these challenges, we propose MatMoE, a dynamic mixed-precision quantization framework for efficient MoE inference. MatMoE features: (1) a Matryoshka MoE PTQ algorithm that enables experts to operate flexibly at multiple precisions within a stringent memory budget; (2) a floor-and-compensation quantization method that eliminates redundant memory accesses in the Matryoshka representation; and (3) a dynamic bit-width allocation strategy that determines the precision for each expert based on their routing scores to satisfy the dynamic latency requirement. We implement our MatMoE system atop this http URL for both CPU and GPU platforms. Extensive evaluations across three modern MoE models and challenging benchmarks demonstrate that MatMoE achieves $4.0\% \sim 18.3\%$ higher accuracy than state-of-the-art static mixed-precision method. Compared to this http URL , MatMoE delivers $1.3\times \sim 3.7\times$ speedup on Raspberry Pi 5, and $1.2\times \sim 1.3\times$ speedup on NVIDIA RTX 4090 and Jetson AGX Orin.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01704 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:46.042Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).