Metraipolis › archive › dc_1afbee49d3c2
Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training — arXiv:2611.03577
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.03577
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:38:18.706Z
- SHA-256 of the archived text
aac3d05fc157d39c22d17857cdaa169253ea32f1917d6c049a01d704e5ad7d73- Stable URL of this page
https://metraipolis.com/documento/dc_1afbee49d3c2
How to cite
Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training — arXiv:2611.03577 — arXiv (en), https://arxiv.org/abs/2611.03577 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1afbee49d3c2 (Metraipolis archive dc_1afbee49d3c2, sha256 aac3d05fc157d39c22d17857cdaa169253ea32f1917d6c049a01d704e5ad7d73)
BibTeX
@misc{dc1afbee49d3c2,
title = {Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training — arXiv:2611.03577},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.03577},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1afbee49d3c2; sha256 aac3d05fc157d39c22d17857cdaa169253ea32f1917d6c049a01d704e5ad7d73},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03577. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. It brings optimizer-in-backward from dense models to batched MoE expert tensors, so the GPU never holds more than one layer of expert gradients. When we fine-tune OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by 19% to 22%. Its updates are bitwise identical to standard training without expert gradient clipping, and held-out loss stays within 0.003 of all baselines. Each step also takes 17% to 23% less time than standard training with global-norm clipping. Updating whole batched tensors rather than individual experts is what keeps Sonar-MoE fast, as it avoids a 20% to 72% slowdown while using the same memory. Sonar-MoE matches optimizer-in-backward over all parameters in memory and speed while still clipping the dense parameters. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Performance (cs.PF). Pagina del lavoro: https://arxiv.org/abs/2611.03577 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:29:48.956Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).