Metraipolis › archive › dc_39342af6beec
ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models — arXiv:2610.30779
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.30779
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-25T08:32:40.215Z
- SHA-256 of the archived text
f40c9877a80792bc035e495cb7279cedceb901800f52e4b2d00735f88ee4607e- Stable URL of this page
https://metraipolis.com/documento/dc_39342af6beec
How to cite
ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models — arXiv:2610.30779 — arXiv (en), https://arxiv.org/abs/2610.30779 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_39342af6beec (Metraipolis archive dc_39342af6beec, sha256 f40c9877a80792bc035e495cb7279cedceb901800f52e4b2d00735f88ee4607e)
BibTeX
@misc{dc39342af6beec,
title = {ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models — arXiv:2610.30779},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.30779},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_39342af6beec; sha256 f40c9877a80792bc035e495cb7279cedceb901800f52e4b2d00735f88ee4607e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30779. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mixture-of-Experts (MoE) large language models reduce computation through sparse activation but still incur substantial storage costs. Mixed-precision post-training quantization (PTQ) reduces these costs by quantizing different model components at different bit widths, but existing methods often struggle to preserve generation quality under tight bit budgets. We identify two key challenges: (1) nonlinear quantization errors at extremely low precision are underestimated when cross-layer propagation is ignored; (2) direct router fine-tuning may cause routing drift, and calibration gains may not generalize to downstream tasks. To address these issues, we propose ProQuant, a weight-only PTQ framework for MoE models. Prune-or-Prioritize Mixed-Precision Allocation (PoP) jointly optimizes expert pruning and bit-width assignment. It allows higher-precision experts to substitute for severely degraded ones while improving bit-budget utilization. Routing-Consistent RMSNorm Finetune (RoCo) adapts expert input representations through mathematically equivalent transformations. It compensates for quantization errors to reduce routing drift. Extensive experiments demonstrate that ProQuant outperforms existing methods across diverse generative tasks. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.30779 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-25T08:31:13.030Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).