Metraipolis › archive › dc_5dbf4631ffa3
ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models — arXiv:2610.01296
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.01296
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T12:04:31.453Z
- SHA-256 of the archived text
749ee247ba7378c1f610ecb149109088256eab093129031f2fd7503de1f831b6- Stable URL of this page
https://metraipolis.com/documento/dc_5dbf4631ffa3
How to cite
ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models — arXiv:2610.01296 — arXiv (en), https://arxiv.org/abs/2610.01296 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5dbf4631ffa3 (Metraipolis archive dc_5dbf4631ffa3, sha256 749ee247ba7378c1f610ecb149109088256eab093129031f2fd7503de1f831b6)
BibTeX
@misc{dc5dbf4631ffa3,
title = {ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models — arXiv:2610.01296},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.01296},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5dbf4631ffa3; sha256 749ee247ba7378c1f610ecb149109088256eab093129031f2fd7503de1f831b6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01296. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) Diffusion Language Models (DLMs) offer flexible parallel decoding and increased model capacity, but their large number of expert parameters incurs substantial computation and storage costs. Existing low-rank MoE compression methods largely rely on static factorization and fixed rank allocation, which overlook the distinctive properties of MoE DLMs. Specifically, we identify two properties: cross-mode non-uniform redundancy, where parameter redundancy and sensitivity to rank truncation vary across the input, output, and expert modes, and token-wise utilization variation, where hot and cold tokens exhibit distinct spectral characteristics and expert activation patterns. To address these challenges, we propose ITC-MoE, an Importance-guided Token-aware Compression framework for MoE DLMs. ITC-MoE consists of two complementary components. First, Importance-guided Adaptive Tucker Compression (IATC) incorporates activation and gradient importance into expert weight transformation, jointly factorizes expert weights across multiple modes, and adaptively allocates ranks under a fixed parameter budget. Second, Token-aware Compensation and Routing (TCR) applies lightweight low-rank compensation to compression-sensitive hot tokens and restricts the candidate expert set for cold tokens with concentrated routing patterns. By jointly adapting compression capacity and inference execution to both parameter redundancy and token-wise variation, ITC-MoE substantially reduces the computation and storage costs of MoE DLMs while preserving their generation quality. For example, on SDAR-30B-A3B-Chat-b32, ITC-MoE maintains an accuracy of 96.33% on MultiArith under a 30% compression budget, while achieving up to a 7.22x end-to-end speedup. The code is publicly available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.01296 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T10:21:27.933Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).