Learning Latent Protein Languages for Autoregressive Generation

Hugging Face Daily Papers Papers
protein-generation latent-language autoregressive-transformer protein-structure esm-2 vqvae scaling-laws

Summary

该论文提出两种学习得到的潜空间蛋白质语言——PLL（基于冻结ESM-2的4,096状态上下文字母表）和SLL（改进的GCP-VQVAE Lite结构分词器），并分别在其上预训练自回归Transformer（PLLM和SLLM），在蛋白质序列与结构生成上相比氨基酸级自回归模型表现出更好的计算扩展性、采样质量以及与AlphaFold2相比约1000倍的推理速度。

Autoregressive transformers remain comparatively weak for protein sequence and structure generation. We study the role of target representation: amino acid tokens encode residue identities without explicit contextual semantics, while backbone coordinates require a discrete representation in our framework. We introduce two learned latent protein languages. Protein Latent Language (PLL) maps sequences to a 4,096-state contextual alphabet built on a frozen ESM-2 encoder, with one token per residue. Structure Latent Language (SLL) adapts GCP-VQVAE Lite with auxiliary sequence and confidence supervision while retaining decoding to backbone coordinates. We separately pretrain autoregressive transformer models on PLL and SLL tokens using next-token prediction, yielding PLLM and SLLM. Under matched downstream sequence training, PLLM has a fitted compute-scaling exponent of 0.038 versus 0.020 for the amino acid autoregressive model. In unconditional sequence generation, PLLM reduces the fraction of samples below a heuristic 1.5-bit residue-composition entropy threshold by 54% relative to the amino acid model across sampling temperatures. For sequence-to-structure prediction, replacing the original GCP-VQVAE Lite tokenizer with SLL reduces best validation perplexity by 34% under matched training. For long proteins, latent-token sampling is approximately 1,000 times faster than MSA-based AlphaFold2 in our measurements. In backbone generation, SLLM compares favorably with other generative models on diversity and novelty. We also observe early signs that using SLLM's internal token confidence for inference-time sampling can improve sequence-to-structure prediction quality beyond a single decoded sample. These results position learned latent protein languages as a promising substrate for autoregressive transformer scaling and inference-time sampling in protein generation.
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:56 PM

Paper page - Learning Latent Protein Languages for Autoregressive Generation

Source: https://huggingface.co/papers/2610.03978

Abstract

Autoregressivetransformersremaincomparativelyweakforproteinsequenceandstructuregeneration.Westudytheroleoftargetrepresentation:aminoacidtokensencoderesidueidentitieswithoutexplicitcontextualsemantics,whilebackbonecoordinatesrequireadiscreterepresentationinourframework.Weintroducetwolearnedlatentproteinlanguages.ProteinLatentLanguage(PLL)mapssequencestoa4,096-statecontextualalphabetbuiltonafrozenESM-2encoder,withonetokenperresidue.StructureLatentLanguage(SLL)adaptsGCP-VQVAELitewithauxiliarysequenceandconfidencesupervisionwhileretainingdecodingtobackbonecoordinates.WeseparatelypretrainautoregressivetransformermodelsonPLLandSLLtokensusingnext-tokenprediction,yieldingPLLMandSLLM.Undermatcheddownstreamsequencetraining,PLLMhasafittedcompute-scalingexponentof0.038versus0.020fortheaminoacidautoregressivemodel.Inunconditionalsequencegeneration,PLLMreducesthefractionofsamplesbelowaheuristic1.5-bitresidue-compositionentropythresholdby54%relativetotheaminoacidmodelacrosssamplingtemperatures.Forsequence-to-structureprediction,replacingtheoriginalGCP-VQVAELitetokenizerwithSLLreducesbestvalidationperplexityby34%undermatchedtraining.Forlongproteins,latent-tokensamplingisapproximately1,000timesfasterthanMSA-basedAlphaFold2inourmeasurements.Inbackbonegeneration,SLLMcomparesfavorablywithothergenerativemodelsondiversityandnovelty.WealsoobserveearlysignsthatusingSLLM’sinternaltokenconfidenceforinference-timesamplingcanimprovesequence-to-structurepredictionqualitybeyondasingledecodedsample.Theseresultspositionlearnedlatentproteinlanguagesasapromisingsubstrateforautoregressivetransformerscalingandinference-timesamplinginproteingeneration.

View arXiv pageView PDFProject pageGitHubAdd to collection

Get this paper in your agent:

hf papers read 2610\.03978

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.03978 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.03978 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.03978 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Does Learning Protein Folding Generalize to Broader Reasoning?

Hugging Face Daily Papers

论文提出 Fold2Reason，一种通过蛋白质折叠数据对语言模型进行后训练的方法，结果表明结构密集的科学数据可以将通用推理基准的宏平均准确率从45.09%提升至48.33%，并显著优于匹配的随机对照实验。

One Model to Loop Them All: Elastic Latent Recurrence for Language Models

arXiv cs.CL

NVIDIA 提出 Elastic Latent Recurrence (ELR) 训练配方，使循环（looped）语言模型在单次循环时达到与同规模非循环模型相当的精度，同时每次额外循环都能以更多推理计算换取精度提升，最终产出可在 1-4 循环间弹性的单一检查点 Koi-4B。

Lapras: Latent Reasoning for Time Series Language Models

arXiv cs.CL

Lapras提出了一种针对时间序列语言模型(TSLM)的后训练框架，通过教师-学生自蒸馏实现连续隐空间推理，在五个时间序列问答基准上平均F1提升最高10.79%，同时生成token减少23.9倍。

Learning to Learn a Language

Hugging Face Daily Papers

This paper introduces the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained exclusively on samples from a synthetic structural causal model, enabling it to learn languages in context with frozen weights despite never seeing real text. Results show near-competitivenatural-language modeling, arithmetic, and compression capabilities below gzip/PPMd, suggesting meta-learning of language rather than memorization.