Metraipolis › archive › dc_c95b933df489

Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment — arXiv:2610.23006

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.23006
License (as archived)
CC0-1.0
Archived
2026-10-19T08:42:22.622Z
SHA-256 of the archived text
73aea18af5338f344f215afaa0e5c78f2d93dd02966a2892ad2ed827507e77b6
Stable URL of this page
https://metraipolis.com/documento/dc_c95b933df489

How to cite

Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment — arXiv:2610.23006 — arXiv (en), https://arxiv.org/abs/2610.23006 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c95b933df489 (Metraipolis archive dc_c95b933df489, sha256 73aea18af5338f344f215afaa0e5c78f2d93dd02966a2892ad2ed827507e77b6)
BibTeX
@misc{dcc95b933df489,
  title = {Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment — arXiv:2610.23006},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.23006},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c95b933df489; sha256 73aea18af5338f344f215afaa0e5c78f2d93dd02966a2892ad2ed827507e77b6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23006. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) models reduce inference computation through sparse expert activation, but their large expert memory footprint often exceeds GPU capacity, making expert offloading necessary for memory-constrained inference. However, existing systems typically handle GPU-missed experts by either transferring them to the GPU, incurring substantial PCIe transfer latency, or executing them on the CPU, making the slower CPU computation the bottleneck. To address these limitations, we present \method, a CPU-GPU collaborative MoE inference system that adaptively assigns these missed experts to the CPU or GPU for efficient execution. By exploiting differences in expert token-loads, \method selectively transfers high-load experts to the GPU while executing low-load experts on the CPU. To minimize inference latency, \method adaptively adjusts expert assignments to balance the CPU computation path against the GPU transfer-and-computation path, allowing both execution paths to complete as early as possible. Extensive experiments on three representative MoE LLMs show that \method improves prefill speed by up to 51.9\% and decode throughput by up to 39.5\% over the strongest baseline across the evaluated batch sizes, with consistent gains under different GPU-memory budgets.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.23006 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:10.051Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).