Metraipolis › archive › dc_b2b02722f804
Cascadia: Resident 975B MoE Inference on Eleven AI PCs — arXiv:2610.07219
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07219
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:18:26.213Z
- SHA-256 of the archived text
6fac225c45b72895918d1271bec79b28a81dd025997c93e09c60e60d20d4ee2f- Stable URL of this page
https://metraipolis.com/documento/dc_b2b02722f804
How to cite
Cascadia: Resident 975B MoE Inference on Eleven AI PCs — arXiv:2610.07219 — arXiv (en), https://arxiv.org/abs/2610.07219 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b2b02722f804 (Metraipolis archive dc_b2b02722f804, sha256 6fac225c45b72895918d1271bec79b28a81dd025997c93e09c60e60d20d4ee2f)
BibTeX
@misc{dcb2b02722f804,
title = {Cascadia: Resident 975B MoE Inference on Eleven AI PCs — arXiv:2610.07219},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07219},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b2b02722f804; sha256 6fac225c45b72895918d1271bec79b28a81dd025997c93e09c60e60d20d4ee2f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07219. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-experts models make nearly trillion-parameter capacity accessible with sparse per-token computation, provided that the serving system can distribute the weights and coordinate their execution. We present Cascadia's resident execution of Inkling, a 975B-total/41B-active-parameter model, on eleven Intel Core Ultra X7 358H AI PCs, each with 64 GB of memory, Arc B390 integrated graphics and gigabit Ethernet. We contribute a custom resident MoE engine that preserves Inkling's routing rules, constructs compressed graphs for OpenVINO's fused iGPU primitives, and coordinates FP16 expert computation with FP32 output restoration. The engine fits six consecutive decoder layers per machine and represents dense feed-forward blocks as all-active expert slices, reducing measured dense-layer call time from approximately 8.1 to 4.5 ms. A streaming pipeline coordinates concurrent generation, while captured-state draft evaluation measures agreement with the deployed numerical path. Paired measurements at fifteen concurrency levels from 1 to 176 streams reach 60.29 aggregate decode tokens/s at 88 streams, with 46.87 tokens/s over the complete serving phases. At fifteen streams, median first-token latency is 6.05 s. Raising the context budget from the 1,024-position default, real prompts of 1k to 64k tokens recover the embedded code in all 19 measured answers, with first-token time growing as $aN+bN^2$ and decode latency growing approximately linearly, both bounded by a single-threaded CPU attention loop rather than by memory, which holds 512k positions per stream. Evaluation on captured fleet states separates the effects of vocabulary selection and weight quantization on draft agreement. Together, these contributions establish an execution and evaluation approach for large sparse models on distributed client systems with shared CPU-GPU memory. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2610.07219 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:24.485Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).