Metraipolis › archive › dc_d6bbc0aea41c

How Small Can We Go? Calibrating Activation-Cache Compression for SAE Training — arXiv:2611.08551

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.08551
License (as archived)
CC0-1.0
Archived
2026-11-09T22:36:05.907Z
SHA-256 of the archived text
05ad612271d5682fb1ed8051427a60ef5dd9c1bf3fa030dac5cf31700865e8b0
Stable URL of this page
https://metraipolis.com/documento/dc_d6bbc0aea41c

How to cite

How Small Can We Go? Calibrating Activation-Cache Compression for SAE Training — arXiv:2611.08551 — arXiv (en), https://arxiv.org/abs/2611.08551 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d6bbc0aea41c (Metraipolis archive dc_d6bbc0aea41c, sha256 05ad612271d5682fb1ed8051427a60ef5dd9c1bf3fa030dac5cf31700865e8b0)
BibTeX
@misc{dcd6bbc0aea41c,
  title = {How Small Can We Go? Calibrating Activation-Cache Compression for SAE Training — arXiv:2611.08551},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08551},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d6bbc0aea41c; sha256 05ad612271d5682fb1ed8051427a60ef5dd9c1bf3fa030dac5cf31700865e8b0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08551. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

To train a sparse autoencoder (SAE), you need a massive cache of model activations. We look into the effects that compression has on an SAE in two ways: we keep the original fp32-trained SAE and pass the compressed cache into the same SAE to see how many feature activations stay the same (the frozen-SAE test), and we retrain an SAE on the compressed cache and see how many of the original features still exist (the retraining test). We found that in the retraining test, there is no clear scale for retained features. On a GPT-2 cache, rounding everything to fp16 keeps 60.6% of features, but changing one value in a million by the smallest step fp32 allows (a change eight orders of magnitude smaller) keeps only 70.5%. Meanwhile, changing the random seed keeps 23.0%. For the frozen-SAE test, the frozen SAE's correlation with the original increases in order of how precise the codecs are, but plain $L_2$ distance gives nearly the same ordering. We also see the same frozen reading correspond to different percentages retained at different sites. This leaves the question of the right measurement. We build one by calibrating: retraining on identical bytes sets the ceiling, and a reseed sets the floor. We then measure the effect of each compressed codec against its own site's floor. With TopK SAEs on GPT-2 small and Pythia-1.4B at research-scale budgets, naïve per-token int4 loses more features than a reseed everywhere we tested, as do PCA and random projection on GPT-2. Standardizing each channel before quantizing brings int4 back above the floor in terms of retained features, at the same raw file size. At our smallest budget, a bf16 forward pass, which some pipelines use to save memory, loses slightly more features than rotated 8-bit storage. Calibrating a site costs four training runs and two evaluation passes.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.08551 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:31:01.954Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).