Metraipolis › archive › dc_338cd9ec64d3
Mixture of Wide Experts: Sparse Routing for Interpretable Native Coordinates — arXiv:2611.03655
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.03655
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T18:38:51.471Z
- SHA-256 of the archived text
8cd102dd3486c3356923e10f2d2d92e57eb8bb3cddeb7cee607e05dbdc07389b- Stable URL of this page
https://metraipolis.com/documento/dc_338cd9ec64d3
How to cite
Mixture of Wide Experts: Sparse Routing for Interpretable Native Coordinates — arXiv:2611.03655 — arXiv (en), https://arxiv.org/abs/2611.03655 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_338cd9ec64d3 (Metraipolis archive dc_338cd9ec64d3, sha256 8cd102dd3486c3356923e10f2d2d92e57eb8bb3cddeb7cee607e05dbdc07389b)
BibTeX
@misc{dc338cd9ec64d3,
title = {Mixture of Wide Experts: Sparse Routing for Interpretable Native Coordinates — arXiv:2611.03655},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.03655},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_338cd9ec64d3; sha256 8cd102dd3486c3356923e10f2d2d92e57eb8bb3cddeb7cee607e05dbdc07389b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03655. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language models often spread a concept across many internal dimensions, making their features difficult to interpret. Sparse autoencoders (SAEs) recover clearer features after training. We ask whether models can instead learn interpretable output coordinates directly. We introduce Mixture of Wide Experts (MoWE), a sparse feed-forward layer in which selected experts read the full representation but write to fixed, disjoint output coordinates. On a controlled task with known hidden causes, this design preserves predictive performance while localizing causes more clearly than dense and standard mixture-of-experts (MoE) models. At approximately matched compute, we train two 19-layer runs of each architecture for 10 billion tokens. MoWE achieves evaluation cross-entropy 2.417 versus 2.452 for dense, while standard MoE reaches 2.22. We measure native-coordinate readability by asking a language-model judge to describe each coordinate from passages where it is strongly active and test that description on new passages. Correcting for descriptions that also fit other coordinates gives a specificity score of 0.523 for MoWE, 0.375 for dense, and 0.268 for standard MoE. TopK SAEs trained on the same FFN updates score 0.489 for dense and 0.556 for MoWE, making native MoWE comparable to the dense SAE. In the first case study, we identify a four-coordinate causal path supporting the model's prediction of degrees after the prompt A right angle is 90. We then copy seven angle-related coordinate values into the model as it processes A rectangle has four, changing its next-token prediction from sides to degrees. In the second, one fixed coordinate set detects hazardous requests and another steers the next token toward refusal. These examples show that MoWE combines sparse conditional computation with coordinates that can be identified and reused directly. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.03655 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T18:32:52.183Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).