Metraipolis › archive › dc_00ac945da8cb

Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors — arXiv:2610.20826

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20826
License (as archived)
CC0-1.0
Archived
2026-10-16T10:20:03.295Z
SHA-256 of the archived text
b5b9fa21bc1faa5dc3a5c54029febd72960fc5742ccf06c90305765b590b14bf
Stable URL of this page
https://metraipolis.com/documento/dc_00ac945da8cb

How to cite

Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors — arXiv:2610.20826 — arXiv (en), https://arxiv.org/abs/2610.20826 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_00ac945da8cb (Metraipolis archive dc_00ac945da8cb, sha256 b5b9fa21bc1faa5dc3a5c54029febd72960fc5742ccf06c90305765b590b14bf)
BibTeX
@misc{dc00ac945da8cb,
  title = {Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors — arXiv:2610.20826},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20826},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_00ac945da8cb; sha256 b5b9fa21bc1faa5dc3a5c54029febd72960fc5742ccf06c90305765b590b14bf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20826. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We investigate whether per-layer, per-task-phase (prompt/reasoning/response) expert-routing statistics in a Mixture-of-Experts (MoE) language model can guide layer-wise mixed-precision quantization of routed-expert FFN tensors (down/gate and up). To attain layerwise routing statistics, we build a routing-capture and profiling pipeline for Qwen-3.6-35B-A3B in this http URL , derive a composite sensitivity score from routing entropy/decisiveness, coefficient of variation (CV), Jaccard/phase-fragility, and top-1 expert strength, and use coordinate descent to tune a layer allocation that respects a fixed average bits-per-weight budget. Notably, the profiling pipeline is applicable to standard top-k MoE architectures. Across independent experimental conditions (about 4.0 bpw flat, about 2.7 bpw steep, and about 2.7 bpw random allocation as a control), reallocation fails to outperform the uniform baseline within each quality tier. In the flat regime, reallocation is statistically neutral (z about 0.9). For the steep tier, the random reallocation performs within 1 SE of the routing-guided reallocation at the same tier, indicating a structural loss at this quality rung caused by mandatory downgrades and Kullback-Leibler Divergence (KLD) tail injection. This aggressive-budget condition shows a decisive loss (z greater than 4 SE) replicated across three independent weight vectors. We argue that the deeper issue is that routing-behavior statistics are an unvalidated proxy for true per-layer quantization cost.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.20826 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:48:02.658Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).