Metraipolis › archive › dc_593ab6072974

Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting — arXiv:2610.22439

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22439
License (as archived)
CC0-1.0
Archived
2026-10-19T08:42:48.021Z
SHA-256 of the archived text
5ef453c38bdd66fc80b558bd753ce66871c300d86961d9c03d0ce22caaad130f
Stable URL of this page
https://metraipolis.com/documento/dc_593ab6072974

How to cite

Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting — arXiv:2610.22439 — arXiv (en), https://arxiv.org/abs/2610.22439 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_593ab6072974 (Metraipolis archive dc_593ab6072974, sha256 5ef453c38bdd66fc80b558bd753ce66871c300d86961d9c03d0ce22caaad130f)
BibTeX
@misc{dc593ab6072974,
  title = {Sparse Autoencoders Can Induce Local Geometry for Mitigating Catastrophic Forgetting — arXiv:2610.22439},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22439},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_593ab6072974; sha256 5ef453c38bdd66fc80b558bd753ce66871c300d86961d9c03d0ce22caaad130f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22439. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Catastrophic forgetting remains a challenge in supervised continual learning, as adapting to new tasks causes representation drift and loss of previously learned information. This raises the question of which representational changes should be constrained while preserving plasticity. We therefore introduce Sparse Autoencoder Geometric Anchoring (SAGA), which converts a Sparse Autoencoder (SAE) trained at each task boundary into local, low-rank constraints on representation drift. Replay examples specify where the constraints apply, while leading right singular directions of the encoder Jacobian specify which changes are penalized. The resulting geometry is input dependent and can be retained without keeping the SAE for subsequent tasks' training. We establish its local relation to latent code matching, define its spectral truncation error, and derive a conditional bound on frozen-readout drift. With replay and snapshot history, the method achieves the highest mean Class-IL accuracy and lowest mean forgetting among the evaluated ER, DER++, ER-ACE, and SGD baselines on three pretrained ViT-B/16 benchmarks, with accuracy gains of 12.15, 4.79, and 3.22 percentage points over the strongest conventional baseline on CIFAR-100, CUB-200, and ImageNet-R, respectively. From scratch results are more mixed, while rank ablations and geometry controls show that SAE-induced low-rank geometry offers useful stability-plasticity trade-offs.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.22439 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:10.105Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).