Metraipolis › archive › dc_eeda94514bc4
Complete or Sparse: A Tale of Two Identifiabilities — arXiv:2610.20851
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20851
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T10:25:00.855Z
- SHA-256 of the archived text
17596b5eddf976e48e0dec16d21571abdb12c7e4c50b64b07490bd9d867f5988- Stable URL of this page
https://metraipolis.com/documento/dc_eeda94514bc4
How to cite
Complete or Sparse: A Tale of Two Identifiabilities — arXiv:2610.20851 — arXiv (en), https://arxiv.org/abs/2610.20851 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eeda94514bc4 (Metraipolis archive dc_eeda94514bc4, sha256 17596b5eddf976e48e0dec16d21571abdb12c7e4c50b64b07490bd9d867f5988)
BibTeX
@misc{dceeda94514bc4,
title = {Complete or Sparse: A Tale of Two Identifiabilities — arXiv:2610.20851},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20851},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eeda94514bc4; sha256 17596b5eddf976e48e0dec16d21571abdb12c7e4c50b64b07490bd9d867f5988},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20851. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Understanding when neural networks learn interpretable representations is central to mechanistic interpretability. We study this through identifiability: when can supervised learning recover the true latent variables underlying the data? We introduce two data generating processes distinguished by their causal direction where either labels cause latents (generative) or latents cause labels (discriminative). When the representation has at least as many dimensions as the latent space, we prove that the optimal encoders are linear in the latents. Conversely, when dimensions are insufficient, identifiability generically fails and encoders become nonlinear. We then show that additional constraints yield recovery of individual latent variables, not just linear mixtures. In the generative setting, non-negative activations and energy efficiency force each neuron to encode exactly one latent, providing a precise account of selective grandmother cell-like responses. In the discriminative setting, when latents are sparse and the true measurement matrix satisfies the spark condition, the learned encoder inherits compressed sensing structure, enabling sparse dictionary learning to perfectly recover individual latents. This constitutes, to our knowledge, the first identifiability result for undercomplete nonlinear independent component analysis. Our framework unifies nonlinear ICA, compressed sensing, and neural network interpretability, delineating exactly when representations transition from nonlinear distortions to linear representations and, under additional constraints or steps, to clean, axis-aligned codes. Theoretical predictions are validated on synthetic benchmarks and on activations drawn from vision models, large language models, and macaque inferotemporal cortex. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.20851 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:48:02.848Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).