Metraipolis › archive › dc_baec4375c325

Interpreting Latent Protein Language Model Features with Geometric Annotations — arXiv:2608.26419

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.26419
License (as archived)
CC0-1.0
Archived
2026-10-17T16:15:48.584Z
SHA-256 of the archived text
406f1d88c7cbe4778cd57124cb9045275dc30e1d23df9f62007b9abf39162e7d
Stable URL of this page
https://metraipolis.com/documento/dc_baec4375c325

How to cite

Interpreting Latent Protein Language Model Features with Geometric Annotations — arXiv:2608.26419 — arXiv (en), https://arxiv.org/abs/2608.26419 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_baec4375c325 (Metraipolis archive dc_baec4375c325, sha256 406f1d88c7cbe4778cd57124cb9045275dc30e1d23df9f62007b9abf39162e7d)
BibTeX
@misc{dcbaec4375c325,
  title = {Interpreting Latent Protein Language Model Features with Geometric Annotations — arXiv:2608.26419},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26419},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_baec4375c325; sha256 406f1d88c7cbe4778cd57124cb9045275dc30e1d23df9f62007b9abf39162e7d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26419. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Protein language models (pLMs) encode information about protein sequences which enable downstream tasks such as structure prediction, but their internal representations are not well understood. Sparse autoencoders (SAEs) provide a promising tool to disentangle latent pLM representations into interpretable features, but existing annotation pipelines largely rely on protein-level annotations derived from database labels and LLM annotations of top activating sequences. Such annotations can overlook the localized residue-level and geometric patterns encoded by sparse features. We introduce an automated and scalable method for interpreting SAE features in ESM-2 by using geometrically inspired features of the protein $\text{C}_\alpha$ backbone. Across ESM-2 8M layers, an FDR-controlled discovery analysis shows that local geometry is significantly associated with many SAE features, with varying levels of predictive strength, expanding coverage beyond database and sequence-based methods. In particular, geometry can distinguish SAE features sharing the same database annotation, revealing substructure within known biological labels. A significant portion of SAE features activate on unannotated metagenomic protein sequences enabling us to use our SAE annotations to better understand these sequences. In addition, ablation experiments at the level of contact prediction show that removing found geometric features shifts ESM-2's predicted contact maps in the direction of the descriptor. This provides a robust method of annotating proteins activated within SAE neurons at a residue level, providing a bridge between mechanistic interpretability and structural biology.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.LG).
Categorie arXiv: Quantitative Methods (q-bio.QM) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.26419 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:34:49.986Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).