Metraipolis › archive › dc_ff993fce300a

PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference — arXiv:2611.03173

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.03173
License (as archived): CC0-1.0
Archived: 2026-11-03T11:13:29.950Z
SHA-256 of the archived text: 566628f1d40ec2761f835d7d50d44f9412eeb8f81a7a961113edc27b1d8627f8
Stable URL of this page: https://metraipolis.com/documento/dc_ff993fce300a

How to cite

PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference — arXiv:2611.03173 — arXiv (en), https://arxiv.org/abs/2611.03173 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff993fce300a (Metraipolis archive dc_ff993fce300a, sha256 566628f1d40ec2761f835d7d50d44f9412eeb8f81a7a961113edc27b1d8627f8)

BibTeX

@misc{dcff993fce300a,
  title = {PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference — arXiv:2611.03173},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03173},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff993fce300a; sha256 566628f1d40ec2761f835d7d50d44f9412eeb8f81a7a961113edc27b1d8627f8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03173. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Mixture-of-Experts (MoE) models are promising for local inference because sparse routing allows the routed-expert pool to reside on SSD while DRAM caches recently used experts. However, under practical DRAM cache capacities, substantial SSD I/O remains on the critical path despite I/O-computation overlap. Token-level dynamic expert precision offers a natural way to reduce this data demand by allocating bits according to expert importance and quantization sensitivity. Nevertheless, treating each bit-width of the same expert as an independent object prevents cross-precision cache reuse, causing overlapping data to be fetched repeatedly. The key insight is that precision should instead denote the refinement depth of one shared expert state. Based on this insight, we present PagedMoE, which stores each expert once using a prefix-compatible tree codebook and bitplane layout. Lower-precision representations are valid prefixes of higher-precision ones, allowing cached data to be reused or extended by fetching only missing bitplanes. PagedMoE selects the required prefix length before I/O using current routing weights and offline expert-specific error profiles, and directly executes cached 1-8-bit prefixes without materializing full-precision experts. At an average routed-expert precision of 2.5 bits, PagedMoE achieves quality close to GPTQ-INT4 on Qwen3.5-397B-A17B and to MXFP4 on most evaluated GLM5.2 tasks. With a 24 GiB expert cache, it reaches 8.89 tokens/s on Qwen3.5, 1.33x the throughput of the same dynamic requests using independent precision objects and 1.56x that of uniform 4-bit execution.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.AR, nuovi lavori del giorno).
Categorie arXiv: Hardware Architecture (cs.AR).
Pagina del lavoro: https://arxiv.org/abs/2611.03173 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:00:22.174Z dalla pagina https://arxiv.org/list/cs.AR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).