Metraipolis › archive › dc_e151cf4ede5c

Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing — arXiv:2609.18131

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18131
License (as archived)
CC0-1.0
Archived
2026-10-15T13:49:53.704Z
SHA-256 of the archived text
71383ec294dff075b748c7463292079dca42db8aa5307cd36e60e29dca592a9d
Stable URL of this page
https://metraipolis.com/documento/dc_e151cf4ede5c

How to cite

Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing — arXiv:2609.18131 — arXiv (en), https://arxiv.org/abs/2609.18131 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e151cf4ede5c (Metraipolis archive dc_e151cf4ede5c, sha256 71383ec294dff075b748c7463292079dca42db8aa5307cd36e60e29dca592a9d)
BibTeX
@misc{dce151cf4ede5c,
  title = {Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing — arXiv:2609.18131},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18131},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e151cf4ede5c; sha256 71383ec294dff075b748c7463292079dca42db8aa5307cd36e60e29dca592a9d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18131. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In this paper, we present a Mixture-of-Experts (MoE) quantization method based on activation entropy. Although quantization reduces memory and computational costs, it can substantially degrade performance. In particular, performance decline is pronounced in quantized MoE models, where individual experts have a small number of parameters that are sensitive to low-bit representation. Considering that MoE operates as an ensemble model with collaborative contributions from routed experts, a significant performance decline of a particular expert due to quantization can harm model performance. Therefore, we propose Colla-Q, a bit-allocation framework to maintain balanced performance across experts through an activation-entropy-based bit-width allocation algorithm. This approach encourages each expert to operate collaboratively in the quantized model, thereby 1) improving the overall MoE performance and 2) reducing the dependence on the calibration dataset. Since uniformly adjusting each expert's performance facilitates robustness and stability of the MoE model, the proposed MoE quantization method can generalize more consistently across different calibration datasets. Our code is available at: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.18131 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:48:24.408Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).