Metraipolis › archive › dc_8501e3cb900b

SharedSAE: One Feature Dictionary Across Language Models — arXiv:2609.04344

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04344
License (as archived)
CC0-1.0
Archived
2026-10-16T22:35:49.901Z
SHA-256 of the archived text
7d056d4265a602f355cc1aeaf92a1e8903b3247047a8bf365f44571981b2c2ad
Stable URL of this page
https://metraipolis.com/documento/dc_8501e3cb900b

How to cite

SharedSAE: One Feature Dictionary Across Language Models — arXiv:2609.04344 — arXiv (en), https://arxiv.org/abs/2609.04344 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8501e3cb900b (Metraipolis archive dc_8501e3cb900b, sha256 7d056d4265a602f355cc1aeaf92a1e8903b3247047a8bf365f44571981b2c2ad)
BibTeX
@misc{dc8501e3cb900b,
  title = {SharedSAE: One Feature Dictionary Across Language Models — arXiv:2609.04344},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04344},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8501e3cb900b; sha256 7d056d4265a602f355cc1aeaf92a1e8903b3247047a8bf365f44571981b2c2ad},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04344. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.04344 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:21:16.313Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).