Metraipolis › archive › dc_cb8ff03029b8

Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation — arXiv:2608.15787

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.15787
License (as archived)
CC0-1.0
Archived
2026-10-18T18:04:30.226Z
SHA-256 of the archived text
592d938f59672ca272b739635b67a393d9cb209ddd62bc1998d277d4dbd22d11
Stable URL of this page
https://metraipolis.com/documento/dc_cb8ff03029b8

How to cite

Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation — arXiv:2608.15787 — arXiv (en), https://arxiv.org/abs/2608.15787 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cb8ff03029b8 (Metraipolis archive dc_cb8ff03029b8, sha256 592d938f59672ca272b739635b67a393d9cb209ddd62bc1998d277d4dbd22d11)
BibTeX
@misc{dccb8ff03029b8,
  title = {Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation — arXiv:2608.15787},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.15787},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cb8ff03029b8; sha256 592d938f59672ca272b739635b67a393d9cb209ddd62bc1998d277d4dbd22d11},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15787. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1.6\times$ as a fraction of block output, while its residual-stream exposure spans $3.2\times$. Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.15787 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:14.052Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).