Metraipolis › archive › dc_93d4e86cc89b
Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility — arXiv:2610.20680
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20680
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T10:19:57.592Z
- SHA-256 of the archived text
b266ea8da00bd3373d4e5754e8b5047a45a99827997643a1ebb9adac5cf79743- Stable URL of this page
https://metraipolis.com/documento/dc_93d4e86cc89b
How to cite
Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility — arXiv:2610.20680 — arXiv (en), https://arxiv.org/abs/2610.20680 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_93d4e86cc89b (Metraipolis archive dc_93d4e86cc89b, sha256 b266ea8da00bd3373d4e5754e8b5047a45a99827997643a1ebb9adac5cf79743)
BibTeX
@misc{dc93d4e86cc89b,
title = {Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility — arXiv:2610.20680},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20680},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_93d4e86cc89b; sha256 b266ea8da00bd3373d4e5754e8b5047a45a99827997643a1ebb9adac5cf79743},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20680. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Sparse autoencoders (SAEs) are typically trained on hundreds of millions to billions of activation tokens, yet little work has tested whether this scale is necessary. We study how much activation data SAEs need by training four SAE architectures on three language models with budgets ranging from 5M tokens to a 500M-token reference. We measure two notions of convergence: functional behavior on standard evaluations and one-to-one reproducibility of learned features. Functional data requirements are strongly metric-dependent: reconstruction often approaches the full-data reference at substantially smaller budgets, whereas feature reproducibility continues to improve with data. For Gemma-2-2B BatchTopK, the fraction of strictly matched features rises from 2\% at 5M tokens to 29\% at 50M. Data selection also matters: random sampling better preserves the original activation distribution and generally favors reconstruction, whereas diversity-based sampling frequently improves AutoInterp. A theoretical analysis separates statistical consistency of subsampled training from the stronger identifiability and optimization conditions required for one-to-one feature recovery. Our results show that SAE training can often be made far more data-efficient when budgets are matched to the goal, and feature-level claims are backed by explicit reproducibility checks, paving the way toward cheaper and more reliable SAE research. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2610.20680 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:48:02.642Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).