Metraipolis › archive › dc_33574c07ccdb

Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests — arXiv:2611.04794

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04794
License (as archived)
CC0-1.0
Archived
2026-11-04T21:45:44.244Z
SHA-256 of the archived text
bc0c23696c43c50bdad18db1757500c10997a2d04b52311cc35021ede643dd1f
Stable URL of this page
https://metraipolis.com/documento/dc_33574c07ccdb

How to cite

Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests — arXiv:2611.04794 — arXiv (en), https://arxiv.org/abs/2611.04794 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_33574c07ccdb (Metraipolis archive dc_33574c07ccdb, sha256 bc0c23696c43c50bdad18db1757500c10997a2d04b52311cc35021ede643dd1f)
BibTeX
@misc{dc33574c07ccdb,
  title = {Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests — arXiv:2611.04794},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04794},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_33574c07ccdb; sha256 bc0c23696c43c50bdad18db1757500c10997a2d04b52311cc35021ede643dd1f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04794. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Expert offloading enables large mixture-of-experts (MoE) language models to run locally, but conventional batching stalls the GPU while missing experts are fetched. A popular remedy, known as prefetching, is to predict what experts will be needed soon and fetch them using idle link bandwidth. Imprecise prediction, however, wastes bandwidth and GPU memory on fetching unnecessary experts. Our key finding is that prefetching can be harmful when there is an expert cache in GPU memory. This is because as the cache includes more necessary experts, expert prediction becomes less precise. Our key insight is that proper request scheduling beats prefetching. Building on this insight, we present Bevy, an MoE inference system based on runahead inference for concurrent requests. A request whose experts are available can run ahead to later layers while other requests wait for fetches, revealing the experts it needs next. Bevy can then fetch the missing experts so identified while computing ready requests, overlapping computation and fetching without predicting expert selections. Running requests separately, however, repeats reads of shared weights in GPU memory and increases compute time. To limit this cost, Bevy's self-balancing scheduler batches ready requests at each layer and prioritizes lagging requests, adapting their grouping as expert availability changes. Across two local platforms, Bevy delivers up to 3.46$\times$ the decode throughput of this http URL and up to 1.55$\times$ that of the fastest prefetching baseline.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04794 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:36:19.387Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).