Metraipolis › archive › dc_17fe6fb45266
A Testable Theory of Atomic Features — arXiv:2610.05794
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.05794
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:25:20.603Z
- SHA-256 of the archived text
e55acf9da26c76c34aade25ac803fb8ad59cd1d8442fbab03ab02a37c77123e9- Stable URL of this page
https://metraipolis.com/documento/dc_17fe6fb45266
How to cite
A Testable Theory of Atomic Features — arXiv:2610.05794 — arXiv, https://arxiv.org/abs/2610.05794 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_17fe6fb45266 (Metraipolis archive dc_17fe6fb45266, sha256 e55acf9da26c76c34aade25ac803fb8ad59cd1d8442fbab03ab02a37c77123e9)
BibTeX
@misc{dc17fe6fb45266,
title = {A Testable Theory of Atomic Features — arXiv:2610.05794},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.05794},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_17fe6fb45266; sha256 e55acf9da26c76c34aade25ac803fb8ad59cd1d8442fbab03ab02a37c77123e9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05794. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We develop and test a theory of language model representations in which there exist atomic features. Our main theoretical insight is that in such a model, sparse dictionaries (e.g., SAEs) of increasing size recover an increasing prefix of the most prevalent atoms in the training data. This "recovery principle" yields three testable predictions: many features in small SAEs are shared by all larger SAEs, SAEs trained on different data share features prevalent in both, and sufficiently large SAEs recover both parent and child features. In contrast to conventional wisdom that SAE features are unstable and "split" as size increases, we find that these predictions hold on SAEs of sizes ranging from 512 to 131,072 trained on two large embedding models. From a theoretical perspective, our results suggest the promise of a scientific theory of representations based on atomic features. Practically, our results suggest the promise of scaling SAEs. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.05794 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:05.630Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).