Metraipolis › archive › dc_804ac1370872

Learning Latent Protein Languages for Autoregressive Generation — arXiv:2610.03978

Source
arXiv
Original
https://arxiv.org/abs/2610.03978
License (as archived)
CC0-1.0
Archived
2026-10-06T05:30:21.281Z
SHA-256 of the archived text
99644dc1cb95fc73ac21c0f8f7ffd0af2651b10d27d40cfdf2b3a500bdba7d57
Stable URL of this page
https://metraipolis.com/documento/dc_804ac1370872

How to cite

Learning Latent Protein Languages for Autoregressive Generation — arXiv:2610.03978 — arXiv, https://arxiv.org/abs/2610.03978 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_804ac1370872 (Metraipolis archive dc_804ac1370872, sha256 99644dc1cb95fc73ac21c0f8f7ffd0af2651b10d27d40cfdf2b3a500bdba7d57)
BibTeX
@misc{dc804ac1370872,
  title = {Learning Latent Protein Languages for Autoregressive Generation — arXiv:2610.03978},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.03978},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_804ac1370872; sha256 99644dc1cb95fc73ac21c0f8f7ffd0af2651b10d27d40cfdf2b3a500bdba7d57},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03978. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Autoregressive transformers remain comparatively weak for protein sequence and structure generation. We study the role of target representation: amino acid tokens encode residue identities without explicit contextual semantics, while backbone coordinates require a discrete representation in our framework. We introduce two learned latent protein languages. Protein Latent Language (PLL) maps sequences to a 4,096-state contextual alphabet built on a frozen ESM-2 encoder, with one token per residue. Structure Latent Language (SLL) adapts GCP-VQVAE Lite with auxiliary sequence and confidence supervision while retaining decoding to backbone coordinates. We separately pretrain autoregressive transformer models on PLL and SLL tokens using next-token prediction, yielding PLLM and SLLM. Under matched downstream sequence training, PLLM has a fitted compute-scaling exponent of 0.038 versus 0.020 for the amino acid autoregressive model. In unconditional sequence generation, PLLM reduces the fraction of samples below a heuristic 1.5-bit residue-composition entropy threshold by 54% relative to the amino acid model across sampling temperatures. For sequence-to-structure prediction, replacing the original GCP-VQVAE Lite tokenizer with SLL reduces best validation perplexity by 34% under matched training. For long proteins, latent-token sampling is approximately 1,000 times faster than MSA-based AlphaFold2 in our measurements. In backbone generation, SLLM compares favorably with other generative models on diversity and novelty. We also observe early signs that using SLLM's internal token confidence for inference-time sampling can improve sequence-to-structure prediction quality beyond a single decoded sample. These results position learned latent protein languages as a promising substrate for autoregressive transformer scaling and inference-time sampling in protein generation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Biomolecules (q-bio.BM).
Pagina del lavoro: https://arxiv.org/abs/2610.03978 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.646Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).