Metraipolis › archive › dc_5c3cc310a129
Instella-MoE Technical Report — arXiv:2609.00791
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.00791
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:33:38.924Z
- SHA-256 of the archived text
0e78e88d28ac7cc8f2e3b263fe99fedd995a4ce2d5882dd9cd634600d16f6e74- Stable URL of this page
https://metraipolis.com/documento/dc_5c3cc310a129
How to cite
Instella-MoE Technical Report — arXiv:2609.00791 — arXiv (en), https://arxiv.org/abs/2609.00791 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5c3cc310a129 (Metraipolis archive dc_5c3cc310a129, sha256 0e78e88d28ac7cc8f2e3b263fe99fedd995a4ce2d5882dd9cd634600d16f6e74)
BibTeX
@misc{dc5c3cc310a129,
title = {Instella-MoE Technical Report — arXiv:2609.00791},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.00791},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5c3cc310a129; sha256 0e78e88d28ac7cc8f2e3b263fe99fedd995a4ce2d5882dd9cd634600d16f6e74},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00791. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In this work, we introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. Instella-MoE combines a sparsely activated MoE design with architectural and system-level innovations, including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity, enabling efficient large-scale training and inference. The model is developed through a multi-stage pipeline comprising pre-training, mid-training, long-context extension, supervised fine-tuning with feedback-driven data curation, direct preference optimization, and reinforcement learning with Multi-Teacher On-Policy Distillation. Instella-MoE achieves an average score of 76.7 across standard pre-training benchmarks, outperforming prior fully open models including OLMo-3-7B, SmolLM3-3B, and OLMoE-1B-7B, while remaining competitive with open-weight MoE and dense baselines at comparable active-parameter scales, including Moonlight-16B-A3B and Qwen3.5-4B. After post-training, our final Think checkpoint achieves an average score of 73.2 across instruction-following, reasoning, math, coding, and chat benchmarks, outperforming both fully open and open-weight models with comparable or larger active parameter counts in our evaluation. To support transparent and reproducible research, we release the complete Instella-MoE model flow, including model weights, training configurations, data mixtures, and training code. Together, these contributions establish Instella-MoE a strong, fully open foundation for efficient, high-performing MoE models and reproducible research. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.00791 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:15:12.820Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).