Metraipolis › archive › dc_fb963a906c4e
Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs — arXiv:2609.00575
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.00575
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:31:57.587Z
- SHA-256 of the archived text
2bf0cc45b9f2492701794b6430e2741ed5e47aa2ccfb0006ada9e5ab072c6956- Stable URL of this page
https://metraipolis.com/documento/dc_fb963a906c4e
How to cite
Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs — arXiv:2609.00575 — arXiv (en), https://arxiv.org/abs/2609.00575 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb963a906c4e (Metraipolis archive dc_fb963a906c4e, sha256 2bf0cc45b9f2492701794b6430e2741ed5e47aa2ccfb0006ada9e5ab072c6956)
BibTeX
@misc{dcfb963a906c4e,
title = {Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs — arXiv:2609.00575},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.00575},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb963a906c4e; sha256 2bf0cc45b9f2492701794b6430e2741ed5e47aa2ccfb0006ada9e5ab072c6956},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00575. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-experts (MoE) architectures scale large language models efficiently, but they demand massive GPU memory. To cope with such demand, models are commonly compressed to reduce their memory footprint. Residual sparsification is a representative compression technique that decomposes each projection matrix of an expert into a shared base matrix and per-expert residual matrix, and then compresses the residuals. Existing sparsification methods compress each residual matrix independently by minimizing its compression error, thereby minimizing the error of each projection matrix. However, our analysis shows that this objective is misaligned with preserving model accuracy after compression. In an expert, the final output is produced through computations coupled across multiple projections and hidden representations. Therefore, even small errors in individual matrices can propagate through hidden representations and projection interactions, leading to large expert output errors and accuracy degradation. To address this misalignment, we propose PARSER, a new residual sparsification method that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. PARSER achieves this by introducing output importance, which measures the actual contribution to the expert output error. Our experiments show that, compared with existing methods, PARSER narrows the accuracy gap to the uncompressed model by 1.41$\times$ on Qwen and 1.44$\times$ on DeepSeek, while achieving the same peak memory reduction. Our code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.00575 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:15:12.618Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).