Metraipolis › archive › dc_f6aa50cc2d5f

TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication — arXiv:2608.27874

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27874
License (as archived)
CC0-1.0
Archived
2026-10-17T10:56:47.783Z
SHA-256 of the archived text
494560ded4a48d47b351d9390f61cc09625540bbb0b68d4092f342480f62ff58
Stable URL of this page
https://metraipolis.com/documento/dc_f6aa50cc2d5f

How to cite

TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication — arXiv:2608.27874 — arXiv (en), https://arxiv.org/abs/2608.27874 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f6aa50cc2d5f (Metraipolis archive dc_f6aa50cc2d5f, sha256 494560ded4a48d47b351d9390f61cc09625540bbb0b68d4092f342480f62ff58)
BibTeX
@misc{dcf6aa50cc2d5f,
  title = {TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication — arXiv:2608.27874},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27874},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f6aa50cc2d5f; sha256 494560ded4a48d47b351d9390f61cc09625540bbb0b68d4092f342480f62ff58},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27874. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hierarchical two-hop dispatch can reduce slow-fabric traffic in expert-parallel Mixture-of-Experts training, but it adds a second collective and an arrival-side operator chain. We present a cost model for screening that trade at the communication-call level, bounded by validation gates that withdraw a capability in code when they fail rather than reporting a caveat. At a reference geometry with 16 groups of 8 ranks, $q=3$, $H=2048$, and 4096 tokens per rank, the corrected effective breakeven hierarchy ratio is 3.98 for the measured PyTorch arrival chain, 1.49 for a hypothetical fused target, and 1.10 at zero implementation overhead. These are ratio-only sensitivity results, not deployment predictions: platform A measures 1.03, platform B has no separated fast/slow measurement, and neither machine measured here reaches the hierarchical regime. Four communication-level corpora pass their gates; a drift probe and the step-level gate fail. The latter failure is enforced in code, so we make no training-throughput prediction. The enabling routing constraint fixes per-token fan-out and per-selected-group quota, while aggregate per-peer counts remain data-dependent. Its measured validation-loss cost is small but nonzero (+0.0034 nats); downstream equivalence is reported with incomplete estimator provenance and is therefore not independently reconstructible from the artifact. Code, calibration constants and the validation gates are at this https URL .

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-31 in cs.DC).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.27874 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:47:50.652Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).