Metraipolis › archive › dc_5cb7a52b7cd4

OOM: Offload-Optimized Full-Parameter MoE Training via Memory-Execution Co-Design — arXiv:2611.11364

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.11364
License (as archived)
CC0-1.0
Archived
2026-11-10T18:45:44.942Z
SHA-256 of the archived text
85aaccfd44fcf65e5903b16cade88bd2dc7534f58011c38f58521ee64739c757
Stable URL of this page
https://metraipolis.com/documento/dc_5cb7a52b7cd4

How to cite

OOM: Offload-Optimized Full-Parameter MoE Training via Memory-Execution Co-Design — arXiv:2611.11364 — arXiv (en), https://arxiv.org/abs/2611.11364 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5cb7a52b7cd4 (Metraipolis archive dc_5cb7a52b7cd4, sha256 85aaccfd44fcf65e5903b16cade88bd2dc7534f58011c38f58521ee64739c757)
BibTeX
@misc{dc5cb7a52b7cd4,
  title = {OOM: Offload-Optimized Full-Parameter MoE Training via Memory-Execution Co-Design — arXiv:2611.11364},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.11364},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5cb7a52b7cd4; sha256 85aaccfd44fcf65e5903b16cade88bd2dc7534f58011c38f58521ee64739c757},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11364. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Offloading model state to host memory and streaming layer weights let a single multi-GPU node train large mixture-of-experts (MoE) models. GPU memory then becomes a performance budget, shared by resident expert state, retained activations, and execution worksets. The best allocation depends on the expert execution path, batch size, and sequence length. We present OOM (Offload-Optimized MoE), a full-parameter MoE training system that co-designs expert execution and memory allocation. For expert-parallel execution, OOM updates resident experts on their owning GPUs and jointly plans state residency and MoE activation retention according to pipeline dependencies. For local execution, OOM bounds GPU worksets through dependency-aware partitioning; in hybrid models, it also retains attention results on the host to avoid costly recomputation and selectively regenerates layer boundaries through cheaper layers to make room. Calibrated cost models guide joint configuration within GPU and host budgets to minimize step time; OOM selects among the configured paths through short executions. On four H100s, OOM trains with 1M-token contexts, whereas FSDP2, ZeRO-Offload, and MegaTrain exhaust GPU memory at 512K. It also trains a 122B-parameter MoE model, on which all three exhaust host memory. On four 24 GB RTX 4090s, it trains 512K-token contexts and 1,024 1K-token sequences per step without gradient accumulation. Across matched-global-batch comparisons, OOM delivers 1.58x and 4.08x geometric-mean (up to 2.14x and 7.77x) throughput over CPU-offloaded FSDP2 on the two platforms while using the least host memory among the evaluated systems -- as little as 1/3.0 of FSDP2's.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.11364 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:38:07.442Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).