Metraipolis › archive › dc_6d2ee79cf968

Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding — arXiv:2610.34721

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.34721
License (as archived)
CC0-1.0
Archived
2026-10-27T07:57:40.256Z
SHA-256 of the archived text
379dcb7086851028702f333202b18272ea384a8f89c145a51870bb22b24bf1c1
Stable URL of this page
https://metraipolis.com/documento/dc_6d2ee79cf968

How to cite

Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding — arXiv:2610.34721 — arXiv (en), https://arxiv.org/abs/2610.34721 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6d2ee79cf968 (Metraipolis archive dc_6d2ee79cf968, sha256 379dcb7086851028702f333202b18272ea384a8f89c145a51870bb22b24bf1c1)
BibTeX
@misc{dc6d2ee79cf968,
  title = {Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding — arXiv:2610.34721},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.34721},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6d2ee79cf968; sha256 379dcb7086851028702f333202b18272ea384a8f89c145a51870bb22b24bf1c1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34721. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When a Mixture-of-Experts (MoE) model does not fit in GPU memory, batch-1 decoding is dominated by copying expert weights from host memory. Cache-aware routing reduces these copies by biasing the router toward resident experts, but combined with asynchronous prefetching it makes the model's outputs depend on transfer timing, so a harness that does not simulate that timing no longer evaluates the deployed model. We propose prefetch-steered routing (PSR): routing conditions on a deterministic logical cache, into which at most n experts predicted by the next layer's router are inserted per layer and token; physical copies affect only latency. On OLMoE-1B-7B with 50% and 25% of experts resident on an RTX 4090, PSR decodes 2.2-2.3x faster than exact routing with LRU and 25-30% faster than the Cache-Prior operating point, with mean benchmark differences of +0.1 and +0.5 points (95% lower bounds -0.8 and -0.3). In greedy generation its lead over Cache-Prior depends on the workload and reverses on HumanEval at 50% residency. On Moonlight-16B-A3B, PSR is 1.9x faster than exact routing with LRU at 37.5% residency and, at the latency of Cache-Prior, significantly better on three of eight benchmarks. On Qwen3.6-35B-A3B, where copies are cheap relative to computation, PSR keeps its advantage over Cache-Prior on the likelihood benchmarks but not its speed. Distillation-trained gates did not help. Speculative decoding with block-level expert selection was slower than PSR with a 1B autoregressive drafter on OLMoE, but with a block drafter on Qwen3.6 it was 2.1-2.9x faster than any autoregressive configuration with at most its GPU memory on the 64 prompts we timed.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.34721 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:30.708Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).