Metraipolis › archive › dc_ec149d4557dd

MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training — arXiv:2608.04407

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.04407
License (as archived)
CC0-1.0
Archived
2026-10-31T11:14:13.028Z
SHA-256 of the archived text
ba6842a7cffbae9c7d312977b5314234233904885693743bcd0b005879950037
Stable URL of this page
https://metraipolis.com/documento/dc_ec149d4557dd

How to cite

MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training — arXiv:2608.04407 — arXiv (en), https://arxiv.org/abs/2608.04407 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ec149d4557dd (Metraipolis archive dc_ec149d4557dd, sha256 ba6842a7cffbae9c7d312977b5314234233904885693743bcd0b005879950037)
BibTeX
@misc{dcec149d4557dd,
  title = {MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training — arXiv:2608.04407},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04407},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ec149d4557dd; sha256 ba6842a7cffbae9c7d312977b5314234233904885693743bcd0b005879950037},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04407. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable. We study this failure in a controlled 110M-parameter nanowhale DeepSeek-style MoE pretraining setting. A SAGE/Sinkhorn hybrid reduces optimizer state from 0.883GB to 0.331GB but degrades evaluation loss to 3.8265, far above the AdamW baselines observed in the same setup (3.58--3.64 across the seeds we study). We show that routed MoE expert matrices are the dominant failure point: their gradients are conditional, temporally varying, and poorly served by stateless Sinkhorn normalization. We propose MESH, a hidden-momentum Sinkhorn update for MoE experts. MESH restores a temporal first-moment signal through the gradient-buffer lifecycle, without storing the expert first moment as optimizer state. MESH is an optional block-preconditioned variant that adds a coarse neuron/block inverse-RMS multiplier. Across ablations, temporal smoothing before matrix normalization is the primary causal ingredient; block/neuron preconditioning can improve the memory-quality frontier, but is not established as universally necessary. In two additional seeds, MESH and MESH-B reduce optimizer-state memory by 62.5\% and peak PyTorch CUDA allocation by about 12.6\% relative to AdamW, with a modest evaluation-loss gap. Full-state diagnostic variants recover AdamW-like performance in ablations, supporting the conclusion that MoE experts need temporal smoothing, but not necessarily full coordinate-wise AdamW state.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.04407 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:02:13.506Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).