Metraipolis › archive › dc_34fdcc3973e6

A Persistent State for Auditable Mixture-of-Experts Routing — arXiv:2609.34634

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34634
License (as archived)
CC0-1.0
Archived
2026-10-07T20:58:00.305Z
SHA-256 of the archived text
cf6ec167fdd3cdb79fe451c92d9796a7decf41db72320da2df5bf679a268657d
Stable URL of this page
https://metraipolis.com/documento/dc_34fdcc3973e6

How to cite

A Persistent State for Auditable Mixture-of-Experts Routing — arXiv:2609.34634 — arXiv (en), https://arxiv.org/abs/2609.34634 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_34fdcc3973e6 (Metraipolis archive dc_34fdcc3973e6, sha256 cf6ec167fdd3cdb79fe451c92d9796a7decf41db72320da2df5bf679a268657d)
BibTeX
@misc{dc34fdcc3973e6,
  title = {A Persistent State for Auditable Mixture-of-Experts Routing — arXiv:2609.34634},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34634},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_34fdcc3973e6; sha256 cf6ec167fdd3cdb79fe451c92d9796a7decf41db72320da2df5bf679a268657d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34634. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mixture-of-Experts (MoE) models repeatedly route tokens to sparse subsets of experts, but conventional routers expose no routing-specific record of how cross-layer influences accumulate. We introduce Scratchpad-Augmented Mixture-of-Experts (SA-MoE), which gives each router access to a low-dimensional persistent state that is not provided to the experts. Learned layerwise writes update this state, and their realized post-update changes exactly decompose the state-mediated contribution to any later routing margin, forming a routing ledger. Across sparsely upcycled SmolLM2- and Gemma-based models and three independent training seeds per architecture, this pathway adds less than 1% analytical forward compute and is strongly used by trained routers: local removal of its router contribution changes the selected Top-2 expert set in 87.6% and 69.9% of decisions, respectively. Relative to a matched latest-write-only control, persistent accumulation increases long-horizon future-routing accessibility by 19.4 and 12.2 percentage points, with positive effects in every seed. More than 90% of absolute ledger contribution comes from non-recent writes in both families, and full-forward suppression of ledger-selected writes changes later routing and output distributions. The ledger is an exact provenance object for the persistent-state pathway, not a complete causal explanation of routing. Sensitivity-aware scores better predict full-forward intervention effects, and post-hoc methods recover related cross-layer attribution without architectural modification. SA-MoE instead makes one routing-specific computational history explicit and directly inspectable within the model's natural forward computation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.34634 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.611Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).