Metraipolis › archive › dc_ab8bc83f2348

Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization — arXiv:2610.20353

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20353
License (as archived)
CC0-1.0
Archived
2026-10-15T14:20:13.398Z
SHA-256 of the archived text
a772081f69e4c6d0f28ca2e57f9cce3eb5ae15dd4aaa58b22eca41f5c8adbc22
Stable URL of this page
https://metraipolis.com/documento/dc_ab8bc83f2348

How to cite

Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization — arXiv:2610.20353 — arXiv (en), https://arxiv.org/abs/2610.20353 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ab8bc83f2348 (Metraipolis archive dc_ab8bc83f2348, sha256 a772081f69e4c6d0f28ca2e57f9cce3eb5ae15dd4aaa58b22eca41f5c8adbc22)
BibTeX
@misc{dcab8bc83f2348,
  title = {Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization — arXiv:2610.20353},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20353},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ab8bc83f2348; sha256 a772081f69e4c6d0f28ca2e57f9cce3eb5ae15dd4aaa58b22eca41f5c8adbc22},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20353. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.20353 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.737Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).