Two-Stage Fine-Tuning for Protein Sequence Generation with Targeted Amino-Acid Composition

# Two-Stage Fine-Tuning for Protein Sequence Generation with Targeted Amino-Acid Composition
Source: [https://arxiv.org/html/2606.27939](https://arxiv.org/html/2606.27939)
Rubén Muñoz\-TafallaAnna María Díaz\-RoviraBertran Miquel\-OliverIsaac Filella\-MerceVíctor Guallar

###### Abstract

Protein language models are standard priors for biological sequence generation, but steering them toward explicit distributional design targets remains largely unexplored\. We study a constrained protein generation problem in which sequences must match a desired amino\-acid \(AA\) composition profile while preserving plausible sequence statistics and diversity\. The motivating application is synthetic feed protein design, where the AA composition of dietary proteins directly determines their nutritional value\. We propose a two\-stage pipeline in which domain\-adaptive fine\-tuning \(FT\) on an in\-domain protein dataset is followed by iterative reward\-weighted FT via reinforcement learning \(RL\) anchored against the FT model as a frozen reference\. We evaluate the pipeline on two AA compositions and find that FT brings the average composition close to the target, while the subsequent RL enforces specific sequence constraints that FT alone cannot satisfy\. We additionally evaluate the design choices of the proposed composition reward term against two baselines and an ablated variant, isolate the contribution of each training stage, and verify that AA composition alignment is achieved without degrading sequence quality\.

protein design, language models, reinforcement learning, composition control, reward\-weighted regression

## 1Introduction

Protein language models \(PLMs\) such as ProtGPT2\(Ferruzet al\.,[2022](https://arxiv.org/html/2606.27939#bib.bib3)\), ProGen2\(Nijkampet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib19)\), and RITA\(Hesslowet al\.,[2022](https://arxiv.org/html/2606.27939#bib.bib14)\)have become standard priors forde novosequence generation\. In most design settings, however, plausibility is not enough: sequences must satisfy an explicit external objective\. Accordingly, PLMs can be steered toward a range of design objectives, including specific family fold\(Madaniet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib12)\), predicted structural confidence\(Stoccoet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib16); Subramanianet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib18)\), enzymatic activity\(Munsamyet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib15); Stoccoet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib16)\), thermostability and binding fitness\(Widatallaet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib17)\), and antimicrobial activity\(Caoet al\.,[2025](https://arxiv.org/html/2606.27939#bib.bib1)\)\. These objectives are pursued through steering strategies that range from prompt\-based conditioning, which uses conditioning tags the model was trained to recognize\(Madaniet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib12); Munsamyet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib15)\), to supervised fine\-tuning \(FT\)\(Madaniet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib12)\)and, more recently, reward\-guided FT via reinforcement learning \(RL\)\(Caoet al\.,[2025](https://arxiv.org/html/2606.27939#bib.bib1); Stoccoet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib16); Subramanianet al\.,[2024](https://arxiv.org/html/2606.27939#bib.bib18)\)\.

These steering strategies frame the design objective as either a categorization \(i\.e\., predicting membership to a predefined class\) or a regression \(i\.e\., optimizing a scalar derived from experimental measurements orin silicoscoring\)\. They are thus not directly suited to design objectives that require matching a distributional target profile, such as a charge profile, a hydrophobicity pattern, or an amino\-acid \(AA\) composition\. Generating proteins with controlled AA compositions could enable applications across biotherapeutics and immunology, biomaterials, and nutrition\. Focusing on nutritional applicability, the design objective of this work is synthetic\-feed protein design, in which candidate proteins should not only match a prescribed AA composition that reflects an idealized nutritional profile but also remain synthesizable and diverse\(Cambra\-Lópezet al\.,[2022](https://arxiv.org/html/2606.27939#bib.bib10); Ravindran,[2013](https://arxiv.org/html/2606.27939#bib.bib11)\)\. The nutritional value of a protein source is largely determined by its AA composition\. In practice, dietary proteins often have AA compositions that do not fully align with the organism’s nutritional requirements, reducing protein digestibility and nitrogen retention\(Emmert and Baker,[1997](https://arxiv.org/html/2606.27939#bib.bib20)\)\. An idealized nutritional profile thus refers to a target AA composition that maximizes coverage of these dietary requirements\.

To this end, we propose a two\-stage pipeline for post\-training alignment under an explicit AA composition objective\. Starting from a pretrained PLM \(ProtGPT2\), the first stage applies domain\-adaptive FT to a subset of natural proteins whose AA composition is closest to the target composition, thereby anchoring the prior near the desired region of sequence space\. The second stage applies iterative reward\-weighted FT via RL, in which at each iteration we generate candidates, score them with a composition reward formulation, filter for length and diversity, and update the policy\. We evaluate the pipeline on two distinct target compositions, including a published reference\(Cambra\-Lópezet al\.,[2022](https://arxiv.org/html/2606.27939#bib.bib10)\)and an in\-domain idealized composition\. Finally, we report the contribution of each training stage, an evaluation of the design choices of our composition reward formulation, and an analysis of preserved sequence quality\.

## 2Related Work

Our work sits within a broader literature on controlled generation for protein sequences\. Recent work has increasingly used reward\-guided FT via RL to steer pretrained PLMs toward design objectives that go beyond sequence plausibility\.Stoccoet al\.\([2024](https://arxiv.org/html/2606.27939#bib.bib16)\)introduces DPO\_pLM, applying direct preference optimization\(Rafailovet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib8)\)to autoregressive PLMs against oracles such as ESMFold pLDDT and the CLEAN enzyme classifier\.Widatallaet al\.\([2024](https://arxiv.org/html/2606.27939#bib.bib17)\)applies DPO to ESM\-IF1 using experimental thermostability measurements, converting scalar stability labels such asΔ​G\\Delta GorΔ​Δ​G\\Delta\\Delta Ginto paired, ranked, or weighted preference objectives\.Caoet al\.\([2025](https://arxiv.org/html/2606.27939#bib.bib1)\)fine\-tunes ProGen2\-XL with proximal policy optimization against a composite reward combining a learned minimum\-inhibitory\-concentration classifier and physicochemical descriptors, designing antimicrobial peptides validated experimentally\.Subramanianet al\.\([2024](https://arxiv.org/html/2606.27939#bib.bib18)\)uses RL on PLMs with structural\-confidence rewards distilled from ESMFold\. These methods align PLMs against external*scalar, ordinal, or categorical*oracles, such as structure\-confidence scores, enzyme\-class predictions, thermostability measurements, or antimicrobial\-activity labels\. Our setting is complementary: the reward directly measures alignment to a target AA composition and is computable analytically from the sequence itself, requiring no external model or experimental measurement\.

Our objective is closest to reward\-weighted and advantage\-weighted regression\(Peters and Schaal,[2007](https://arxiv.org/html/2606.27939#bib.bib7); Penget al\.,[2019](https://arxiv.org/html/2606.27939#bib.bib6)\), with the frozen reference model as a KL trust region in the spirit of DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib8)\)and PPO\. Rather than constructing preference pairs as in DPO, we use a softmax reward\-weighting on the per\-batch candidate pool, which naturally handles continuous\-valued rewards without explicit pair construction\. The loss is given in §[3\.3](https://arxiv.org/html/2606.27939#S3.SS3)\.

## 3Methods

### 3\.1Problem Formulation

Letp​\(s\)∈𝒮20p\(s\)\\in\\mathcal\{S\}\_\{20\}denote the empirical AA frequency vector of a protein sequencess, where𝒮20=\{x∈ℝ≥120:∑ixi=1\}\\mathcal\{S\}\_\{20\}=\\\{x\\in\\mathbb\{R\}\_\{\\geq 1\}^\{20\}:\\sum\_\{i\}x\_\{i\}=1\\\}is the probability simplex over the twenty canonical AA, and letq∈𝒮20q\\in\\mathcal\{S\}\_\{20\}be a target composition\. We want to adapt a pretrained PLM so that it generates sequences satisfyingp​\(s\)≈qp\(s\)\\approx qwhile preserving sequence plausibility, valid length, and pairwise diversity\.

### 3\.2First Stage: Base Model and Domain\-Adaptive FT

Pretraining on in\-domain data is a standard recipe for domain adaptation\(Gururanganet al\.,[2020](https://arxiv.org/html/2606.27939#bib.bib4)\)\. We use it as a composition\-conditioned adaptation, anchoring \(πref\\pi\_\{\\text\{ref\}\}\) near a region of sequence space that is both biologically plausible and compositionally similar toqq\. Starting from the UniProtKB/TrEMBL release\(The UniProt Consortium,[2023](https://arxiv.org/html/2606.27939#bib.bib21)\)\(the unreviewed, automatically annotated portion of UniProt; downloaded in FASTA format from the UniProt Consortium FTP repository,∼\\sim2\.5×1082\.5\{\\times\}10^\{8\}sequences\), we apply three filters: \(i\) a length filter100100\-500500AA \(retains∼\\sim1\.8×1081\.8\{\\times\}10^\{8\}sequences\); \(ii\) a cosine\-similarity filter againstqqat a threshold≥0\.95\\geq 0\.95, which retains∼\\sim2\.5×1052\.5\{\\times\}10^\{5\}sequences whose own composition resembles the target composition; and \(iii\) a sequence\-identity filter at<70%<70\\%pairwise identity to remove near\-duplicates and avoid bias toward over\-represented families, leaving∼\\sim1\.0×1051\.0\{\\times\}10^\{5\}sequences\. The same pipeline is applied to two target compositions \(qAq\_\{\\text\{A\}\}andqBq\_\{\\text\{B\}\}; §[4\.1](https://arxiv.org/html/2606.27939#S4.SS1)\), yielding two distinct FT datasets\. The resulting FT dataset contains natural proteins that are both biologically plausible and compositionally close toqq, providing a training signal that biases the base model toward the target composition before any reward\-guided RL optimization\.

The base model is ProtGPT2\(Ferruzet al\.,[2022](https://arxiv.org/html/2606.27939#bib.bib3)\)\. We perform causal\-LM domain\-adaptive FT on the FT dataset and take the resulting checkpoint \(FT prior\) as the frozen reference policyπref\\pi\_\{\\text\{ref\}\}for all subsequent RL runs \(hyperparameters in App\.[A](https://arxiv.org/html/2606.27939#A1)\)\.

### 3\.3Second Stage: Reward\-Weighted RL

Given the frozen reference policyπref\\pi\_\{\\text\{ref\}\}, we optimize a trainable policyπθ\\pi\_\{\\theta\}with a reward\-weighted log\-ratio objective\. For a batch of sampled sequences\{si\}\\\{s\_\{i\}\\\}, we compute scalar rewardsr​\(si\)r\(s\_\{i\}\), standardize and clip them within the batch to obtainr~i\\tilde\{r\}\_\{i\}, and convert them to weightswi=softmax​\(r~i\)w\_\{i\}=\\mathrm\{softmax\}\(\\tilde\{r\}\_\{i\}\), so that sequences with higher rewards contribute more to the update\. The objective is

ℒ​\(θ\)=−∑iwi​η​\(log⁡πθ​\(si\)−log⁡πref​\(si\)\)\+λKL​KL​\(πθ∥πref\)\.\\begin\{split\}\\mathcal\{L\}\(\\theta\)=\\;&\-\\sum\_\{i\}w\_\{i\}\\,\\eta\\bigl\(\\log\\pi\_\{\\theta\}\(s\_\{i\}\)\-\\log\\pi\_\{\\text\{ref\}\}\(s\_\{i\}\)\\bigr\)\\;\\\\ &\+\\;\\lambda\_\{\\text\{KL\}\}\\,\\mathrm\{KL\}\\\!\\bigl\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\text\{ref\}\}\\bigr\)\.\\end\{split\}\(1\)whereη\\etais a log\-ratio scale factor\. This is related to reward\-weighted and advantage\-weighted regression\(Peters and Schaal,[2007](https://arxiv.org/html/2606.27939#bib.bib7); Penget al\.,[2019](https://arxiv.org/html/2606.27939#bib.bib6)\), withπref\\pi\_\{\\text\{ref\}\}playing a role analogous to that in DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib8)\)\. The Kullback\-Leibler termKL​\(πθ∥πref\)\\mathrm\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)measures how farπθ\\pi\_\{\\theta\}has drifted fromπref\\pi\_\{\\text\{ref\}\}\. Penalizing it acts as a trust region that preventsπθ\\pi\_\{\\theta\}from exploiting reward shortcuts at the expense of sequence plausibility\.

Each iterationtt: \(i\) generate a candidate pool of sequences with stochastic decoding; \(ii\) generate a parallel*diversity\-pulse*pool with hotter decoding \(higher temperature andtop​\-​p\\mathrm\{top\\text\{\-\}p\}; §[H](https://arxiv.org/html/2606.27939#A8)\); \(iii\) score both withr​\(s\)r\(s\); \(iv\) filter sequences by length, de\-duplication, and pairwise identity below0\.850\.85using MMseqs2\(Steinegger and Söding,[2017](https://arxiv.org/html/2606.27939#bib.bib9)\); \(v\) re\-inject a fraction of the diversity\-pulse pool into the candidate pool, with the re\-injection fraction increasing across training to counteract late\-iteration mode collapse; \(vi\) construct reward\-weighted batches and updateπθ\\pi\_\{\\theta\}under Eq\.[1](https://arxiv.org/html/2606.27939#S3.E1)\.

The reward combines a composition term and a length term,

r​\(s\)=wc​Comp​\(s,q\)\+wl​Len​\(s\),r\(s\)=w\_\{c\}\\,\\mathrm\{Comp\}\(s,q\)\+w\_\{l\}\\,\\mathrm\{Len\}\(s\),\(2\)with static weights\(wc,wl\)=\(0\.97,0\.03\)\(w\_\{c\},w\_\{l\}\)=\(0\.97,0\.03\)across all runs\. The composition term is the primary signal, while the length term prevents collapse to extremes, motivating the asymmetric weighting\. These weights were chosen as a working default after a small exploration\. No controlled sweep was performed, so we make no claim of optimality\.Len​\(s\)\\mathrm\{Len\}\(s\)is a piecewise\-linear shaping term that is non\-zero on\[70,400\]\[70,400\]AA with a peak plateau on\[110,250\]\[110,250\]\(App\.[B](https://arxiv.org/html/2606.27939#A2)\)\. This range used at training does not have to coincide with the FT dataset length filter\[100,500\]\[100,500\]AA\. The length term and weights are identical across all reward variants analyzed in this work, and onlyComp​\(s,q\)\\mathrm\{Comp\}\(s,q\)differs\.

We propose a*differentiated*composition term as our primaryComp​\(s,q\)\\mathrm\{Comp\}\(s,q\)term and compare it against one ablation and two baselines at matched compute to assess the contribution of its design choices\. The*differentiated*composition term uses an asymmetric per\-residue kernel \(penalizing deficits in essential AA, those that cannot be synthesized by the organism and must be obtained through diet, more harshly than excesses\), two residue pools whose members are treated as biochemically interchangeable \(a sulfur pool \(Met/Cys\) and an aromatic\-precursor pool \(Phe/Tyr\)\), and a zero\-target residue amplifier \(a term that up\-weights residues whose target frequency is zero\)\. As an ablation, we evaluate a*symmetric*variant of the*differentiated*term in which the asymmetric per\-residue weighting is replaced by uniform absolute deviations\. As baselines, we additionally evaluate a*cosine*similarity term and a*global\-deviation*L1L\_\{1\}composition term, both of which lack the per\-residue structure of the*differentiated*composition term\. Full formulas and hyperparameters App\.[B](https://arxiv.org/html/2606.27939#A2)\.

Each composition term has a sharpness coefficientβ\\betainside an outerexp⁡\(−β​\[⋅\]\)\\exp\(\-\\beta\\,\[\\cdot\]\)that maps the per\-sequence composition error to a bounded reward, withβ\\betaramping during training to progressively sharpen the reward signal\. A fixed reference valueβref\\beta\_\{\\text\{ref\}\}is used at evaluation, applied to the*differentiated*term so that all variants are compared on a common scoring function \(full per\-variant ramps in App\.[B](https://arxiv.org/html/2606.27939#A2); loop hyperparameters in App\.[A](https://arxiv.org/html/2606.27939#A1)\)\.

## 4Experimental setup

### 4\.1Target AA Compositions

qAq\_\{\\text\{A\}\}\(primary\): an experimentally refined poultry\-feed AA composition provided by a project partner \(details withheld for proprietary reasons\)\. It includes two low\-frequency residues \(one at exactly zero\)\. We anonymized the residue labels asaa1,…,aa20\\mathrm\{aa\}\_\{1\},\\ldots,\\mathrm\{aa\}\_\{20\}throughout \(sorted by descending target frequency\)\.

qBq\_\{\\text\{B\}\}\(published reference\): a poultry\-feed AA composition derived fromCambra\-Lópezet al\.\([2022](https://arxiv.org/html/2606.27939#bib.bib10)\), with twenty non\-zero target frequencies\. Used as a generalization probe\.

Table 1:Pipeline comparison on the primary targetqAq\_\{\\text\{A\}\}and the published reference targetqBq\_\{\\text\{B\}\}\. All values are pool means at the run’s best iteration \(selected by mean composition score atβref=20\\beta\_\{\\text\{ref\}\}\{=\}20\)\. The “RL \(n=⋅n\{=\}\\cdot\)” rows aggregate all seeds of the*differentiated*composition term on the corresponding target \(mean±\\,\\pm\\,std across seeds for JSD and Comp\.\), and the “RL \(best run\)” rows report the single seed with the lowest pool\-mean JSD on that target\.StageTargetComp\.↑\\uparrowJSD↓\\downarrowN±30↑N\_\{\\pm 30\}\\uparrowBase ProtGPT2qAq\_\{\\text\{A\}\}0\.0070\.2473\.70\+ Domain\-adapt\. FTqAq\_\{\\text\{A\}\}0\.1260\.0599\.17\+ RL \(n=30n\{=\}30\)qAq\_\{\\text\{A\}\}0\.397±0\.2100\.397\{\\pm\}0\.2100\.032±0\.0200\.032\{\\pm\}0\.02012\.34\+ RL \(best run\)qAq\_\{\\text\{A\}\}0\.8220\.004417\.40Base ProtGPT2qBq\_\{\\text\{B\}\}0\.0540\.2023\.71\+ Domain\-adapt\. FTqBq\_\{\\text\{B\}\}0\.2070\.0518\.25\+ RL \(n=10n\{=\}10\)qBq\_\{\\text\{B\}\}0\.566±0\.1230\.566\{\\pm\}0\.1230\.021±0\.0080\.021\{\\pm\}0\.00812\.21\+ RL \(best run\)qBq\_\{\\text\{B\}\}0\.8300\.000820\.00
### 4\.2Runs

We run all four composition term variants of §[3\.3](https://arxiv.org/html/2606.27939#S3.SS3)on each target at matched compute and shared seeds\. OnqAq\_\{\\text\{A\}\}, each variant is run on3030seeds; onqBq\_\{\\text\{B\}\}, each variant is run on1010seeds\. All other recipe knobs \(number of iterations, optimizer, KL schedule,β\\betaramp, candidates per iteration, and the length and identity filters\) are identical across variants and across targets\. This gives4×30\+4×10=1604\{\\times\}30\+4\{\\times\}10=160RL runs in total, plus base ProtGPT2 \(no further training\) and a domain\-adaptive FT checkpoint per target composition, evaluated as four baseline rows in Table[1](https://arxiv.org/html/2606.27939#S4.T1)\.

### 4\.3Metrics

We evaluate each run at the iteration whose candidate pool has the highest mean composition score, recomputed at fixedβref=20\\beta\_\{\\text\{ref\}\}\{=\}20through the*differentiated*composition term \(App\.[B](https://arxiv.org/html/2606.27939#A2)\), so that selection is independent of each run’sβ\\betaschedule\.

We report the*Jensen\-Shannon divergence*JSD​\(p,q\)\\mathrm\{JSD\}\(p,q\)between the empirical and target composition, the*tolerance count*N±30​\(p,q\)N\_\{\\pm 30\}\(p,q\), defined as the number of residues whose empirical frequency falls within a30%30\\%relative window of their target frequency, and the composition score atβref\\beta\_\{\\text\{ref\}\}\. All three are computed per sequence and then averaged over the candidate pool of the selected iteration\. We therefore report pool means for each run, and, for the multi\-seed ”RL” rows, mean±\\,\\pm\\,std across seeds of these per\-run pool means\. Pool sizes range from∼\\sim700700to∼\\sim20002000sequences depending on iteration, after length and identity filtering\. We additionally report three indicators measuring the fraction of sequences in the candidate pool that satisfy a given sequence constraint:*essential\-residue coverage*\(fraction of sequences in which the ten essential AA each reach at least half of their target frequency\),*pool tolerance*\(both interchangeable pools within±30%\\pm 30\\%of their pool target\), and*low\-target compliance*\(at most two occurrences of any residue whose target frequency is at or near zero\)\. The low\-target compliance metric is reported only onqAq\_\{\\text\{A\}\}becauseqBq\_\{\\text\{B\}\}has no zero or near\-zero frequency residues\.

Sequence quality \(§[5\.4](https://arxiv.org/html/2606.27939#S5.SS4)\) is summarized by NetSolP\-predicted solubility\(Thumuluriet al\.,[2022](https://arxiv.org/html/2606.27939#bib.bib13)\), base ProtGPT2 log\-PPL, ESM\-2 pPPL, mean length, and intra\-pool similarity\. Per\-variant means use95%95\\%percentile bootstrap CIs over the seed count\.

Table 2:Pipeline comparison of sequence constraint indicators on the primary targetqAq\_\{A\}and the published reference targetqBq\_\{B\}fraction of sequences in the candidate pool satisfying each indicator\)\. The “RL \(n=⋅n\{=\}\\cdot\)” rows aggregate all seeds of the corresponding target, and the “RL \(best run\)” rows report the single seed with the lowest pool\-mean JSD on that target\.

## 5Results

### 5\.1Domain Adaptation Moves the Base Model Toward the Target AA Composition

We first examine the effect of domain\-adaptive FT before any reward\-weighted RL is applied\. Independent FT runs on the respective target\-specific FT datasets produce a large shift toward their target compositions \(Table[1](https://arxiv.org/html/2606.27939#S4.T1)\)\.

OnqAq\_\{\\text\{A\}\}, held\-out perplexity on the FT eval split \(a held\-out subset of the composition\-filtered UniProt sequences, §[3\.2](https://arxiv.org/html/2606.27939#S3.SS2), App\.[A](https://arxiv.org/html/2606.27939#A1)\) drops by an order of magnitude \(5570\.6→507\.35570\.6\\to 507\.3\)\. Concurrently, JSD drops from0\.2470\.247to0\.0590\.059bits, the tolerance countN±30N\_\{\\pm 30\}rises from3\.703\.70to9\.179\.17, and the composition score rises from0\.0070\.007to0\.1260\.126\(Table[1](https://arxiv.org/html/2606.27939#S4.T1)\)\. Because this eval split is drawn from the composition\-filtered UniProt subset that conditions the FT \(§[3\.2](https://arxiv.org/html/2606.27939#S3.SS2)\), the held\-out perplexity primarily reflects domain adaptation rather than generic protein\-likeness\. For an independent protein\-likeness proxy, we use ESM\-2 \(150M\) pseudo\-perplexity \(pPPL\)\(Linet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib5)\), which we report in full in §[5\.4](https://arxiv.org/html/2606.27939#S5.SS4)\(Table[3](https://arxiv.org/html/2606.27939#S5.T3)\) and which shows that FT is slightly worse than base ProtGPT2\. OnqBq\_\{\\text\{B\}\}the picture is qualitatively the same \(Table[1](https://arxiv.org/html/2606.27939#S4.T1)\): JSD drops from0\.2020\.202to0\.0510\.051,N±30N\_\{\\pm 30\}rises from3\.713\.71to8\.258\.25, and the composition score from0\.0540\.054to0\.2070\.207\.

One qualification follows\. The average composition is closer toqq, but sequence constraint indicators remain weak after FT \(essential\-residue coverage0\.2120\.212, pool tolerance0\.2760\.276, low\-target compliance0\.0980\.098onqAq\_\{\\text\{A\}\}, with analogous gaps onqBq\_\{\\text\{B\}\}; Table[2](https://arxiv.org/html/2606.27939#S4.T2)\)\. This is expected, as the FT dataset is built from natural proteins selected by AA composition cosine similarity \(§[3\.2](https://arxiv.org/html/2606.27939#S3.SS2)\), and the unconstrained likelihood objective cannot enforce sharp residue constraints that diverge from typical natural composition\. The reward\-weighted RL stage closes exactly this gap \(§[5\.2](https://arxiv.org/html/2606.27939#S5.SS2)\)\. Whether the FT stage is necessary for this gap closure is examined as an ablation in §[5\.5\.2](https://arxiv.org/html/2606.27939#S5.SS5.SSS2)\.

### 5\.2Reward\-Weighted RL Closes Sequence Constraints Gap

Having established that domain\-adaptive FT improves average composition but leaves sequence constraints largely unsatisfied, we now show that adding the RL stage on top of FT closes the remaining gap\. Using the*differentiated*term onqAq\_\{\\text\{A\}\}results in a mean JSD of0\.032±0\.0200\.032\\pm 0\.020across seeds \(n=30n=30\), compared to0\.0590\.059for FT alone, that is, a roughly2×2\\timesreduction \(Table[1](https://arxiv.org/html/2606.27939#S4.T1)\)\. The seed distribution is right\-skewed: the best run reachesJSD=0\.0044\\mathrm\{JSD\}=0\.0044a composition score of0\.8220\.822,N±30=17\.4N\_\{\\pm 30\}=17\.4\(Table[1](https://arxiv.org/html/2606.27939#S4.T1)\), and all sequence constraints indicators \(low\-target compliance, essential\-residue coverage, pool tolerance\) are saturated at1\.001\.00, while FT and base ProtGPT2 leave most sequences out of compliance \(Table[2](https://arxiv.org/html/2606.27939#S4.T2)\)\. The mean\-across\-seeds row is intermediate, with typical seeds satisfying roughly half of each sequence constraint indicator\. Five of the3030seeds reachJSD<0\.01\\mathrm\{JSD\}<0\.01\. The best\-seed numbers are operationally relevant because the best runs are the ones whose candidates will be selected for synthesis\. The full top\-5 by JSD and by composition score is reported in App\.[C](https://arxiv.org/html/2606.27939#A3)\. OnqBq\_\{\\text\{B\}\}the*differentiated*term reaches a meanJSD=0\.021±0\.008\\mathrm\{JSD\}=0\.021\\pm 0\.008\(n=10\)\. The best run achievesJSD=0\.0008\\mathrm\{JSD\}=0\.0008, a composition score of 0\.830,N±30=20\.0N\_\{\\pm 30\}=20\.0and all applicable sequence constraints indicators saturated at 1\.00 \(Table[1](https://arxiv.org/html/2606.27939#S4.T1)\)\. TheqBq\_\{\\text\{B\}\}best run is closer to its target than theqAq\_\{\\text\{A\}\}best run, butqBq\_\{\\text\{B\}\}is a strictly easier task: it has no zero\-frequency residues, so the zero\-target residue amplifier that dominates the*differentiated*composition term onqAq\_\{\\text\{A\}\}does not apply\. The two ”best” numbers are therefore not directly comparable\.

The fixed\-βref\\beta\_\{\\text\{ref\}\}composition score rises monotonically across iterations \(Fig\.[1](https://arxiv.org/html/2606.27939#S5.F1), bottom\) for the median of the top six seeds of every composition term variant\. Whether the RL gains over FT can be matched by best\-of\-N selection from the FT prior alone is examined as an ablation in §[5\.5\.3](https://arxiv.org/html/2606.27939#S5.SS5.SSS3)\.

![Refer to caption](https://arxiv.org/html/2606.27939v1/x1.png)

Figure 1:Training dynamics onqAq\_\{\\text\{A\}\}\. Each curve is the median across the top six seeds per composition term variant, with the IQR shaded\. \(Top\) Mean JSD against the target across iterations\. \(Bottom\) Composition score evaluated at a fixed reference temperatureβref=20\\beta\_\{\\text\{ref\}\}\{=\}20using the*differentiated*composition term for all four variants\.
### 5\.3Per\-Residue Calibration

Figure[2](https://arxiv.org/html/2606.27939#S5.F2)shows the per\-residue composition calibration of the best RL run relative to the FT prior and base ProtGPT2, revealing where each training stage contributes most\. The top panel shows observed vs\. target counts for every residue \(anonymized asaa1,…,aa20\\mathrm\{aa\}\_\{1\},\\ldots,\\mathrm\{aa\}\_\{20\}, sorted by descending target frequency\)\. The highest\-frequency target residues are already well\-matched after FT, with RL providing only modest additional sharpening\. Intermediate\-frequency residues \(positions∼\\sim5\-15\) are where RL contributes the most, often pulling absolute frequency residuals\|pi−qi\|\|p\_\{i\}\-q\_\{i\}\|from∼0\.02\\sim 0\.02after FT down to<0\.005<0\.005\. Residues whose target frequency deviates substantially from the natural\-protein average, including the lowest\-target residues, are barely moved by FT and are brought close to their target by the RL stage\.

![Refer to caption](https://arxiv.org/html/2606.27939v1/x2.png)

Figure 2:Per\-residue calibration onqAq\_\{\\text\{A\}\}\(residues anonymized asaa1,…,aa20\\mathrm\{aa\}\_\{1\},\\ldots,\\mathrm\{aa\}\_\{20\}, sorted by descendingqq\)\. Counts are pool\-mean frequenciespip\_\{i\}rescaled to a common reference lengthL=292L\{=\}292AA \(the rounded mean sequence length of the best\-RL pool\)\. \(Top\) Target counts \(blue\) vs\. best\-RL counts \(turquoise\); Domain\-adaptive FT and base ProtGPT2 are overlaid as dashed and dotted lines\. \(Middle\) Signed count residual \(observed vs\. target\) for the best\-RL run\. \(Bottom\) Signed relative residual\(p−q\)/q×100%\(p\-q\)/q\\times 100\\%for the best\-RL run \(length\-independent\), theq=0q\{=\}0residue is marked n/a\.
### 5\.4Composition Alignment Preserves Sequence Quality

We next ask whether target composition alignment comes at the cost of sequence plausibility\. Table[4](https://arxiv.org/html/2606.27939#S5.T4)summarizes three sequence\-level quality indicators across all RL runs on both target compositions: predicted solubility \(NetSolP\), which is relevant because high solubility is a prerequisite for protein digestibility, and two protein\-likeness scores, base ProtGPT2 log\-PPL \(autoregressive, under the original pretrained model\) and ESM\-2 pPPL \(masked\-LM, from a different model family\)\.

Mean NetSolP solubility is0\.582±0\.0300\.582\\pm 0\.030onqAq\_\{\\text\{A\}\}and0\.628±0\.0530\.628\\pm 0\.053onqBq\_\{\\text\{B\}\}, both above the NetSolP0\.50\.5decision threshold of the original NetSolP study, with individual sequences exceeding0\.90\.9on both targets\. NetSolP is itself a predictor, so these scores are an*in\-silico*sanity check, not a wet\-lab claim\. The four composition terms are within0\.040\.04of each other in mean solubility onqAq\_\{\\text\{A\}\}\(0\.5600\.560\-0\.5980\.598; Table[7](https://arxiv.org/html/2606.27939#S5.T7)\) and within0\.100\.10onqBq\_\{\\text\{B\}\}\(0\.5890\.589\-0\.6930\.693; per\-variantqBq\_\{\\text\{B\}\}values in Table[10](https://arxiv.org/html/2606.27939#A4.T10), App\.[D](https://arxiv.org/html/2606.27939#A4)\)\.

Table 3:ESM\-2 \(150M\) pseudo\-perplexity \(pPPL\) as a protein\-likeliness measure across pipeline stages, composition term variants, and target compositions; mean±\\pmstd over up ton=200n\{=\}200unique sequences per condition; lower is better\. RL rows use the best iteration of one representative seed per variant\. ESM\-2 pPPL is length\-dependent, so we also report mean sequence lengthL¯\\bar\{L\}\.Table 4:Sequence\-quality summary, mean±\\pmstd across all RL runs per target\. NetSolP solubility is a predicted score in\[0,1\]\[0,1\]\(higher is more soluble\)\.*base log\-PPL*is the mean log\-perplexity of the base ProtGPT2 policy\. ESM\-2 \(150M\) pseudo\-perplexity \(pPPL\) is computed on up ton=200n\{=\}200unique sequences per RL composition variant; the entry summarizes mean±\\pmstd across the four primary variants\. The*iid floor*column reports the same three metrics onn=100n\{=\}100random sequences sampled iid fromqAq\_\{\\text\{A\}\}atL=400L\{=\}400, as a composition\-only baseline\.ESM\-2 pPPL increases monotonically from base ProtGPT2 \(mean4\.484\.48on both targets, as expected for an unconditional baseline\) to the domain\-adapted FT prior \(5\.745\.74\-6\.046\.04\) to the RL policies \(6\.726\.72\-16\.6516\.65, target\- and composition\-dependent\) \(Table[3](https://arxiv.org/html/2606.27939#S5.T3)\)\. For reference, ESM\-2 pPPL on natural sequences typically falls in the single\-digit range\(Linet al\.,[2023](https://arxiv.org/html/2606.27939#bib.bib5)\), so the RL policies sit one tier above, indicating a measurable but moderate plausibility cost\. This cost is not uniform across composition terms: the*symmetric*composition term in particular almost triples ESM\-2 pPPL relative to the FT prior onqAq\_\{\\text\{A\}\}, which we interpret as the cost of steering the policy toward a non\-natural composition without the biological weighting of the*differentiated*term\. Notably, the*differentiated*term \(mean ESM\-2 pPPL9\.419\.41onqAq\_\{\\text\{A\}\}\) incurs the smallest increase among the composition variants, suggesting that its biological weighting partially mitigates this plausibility cost\. To stress\-test the plausibility of the most composition\-aligned sequences, we additionally score the top\-3030sequences from each policy ranked by composition score \(i\.e\. the most aggressive on\-target sequences each policy generates\) \(Table[15](https://arxiv.org/html/2606.27939#A6.T15)in App\.[F](https://arxiv.org/html/2606.27939#A6)\)\. Even on this worst\-case slice, no policy collapses to a degenerate high\-ESM\-2 pPPL regime, and the*differentiated*variant retains its small\-fluency\-penalty advantage over the other smooth\-max variant onqAq\_\{\\text\{A\}\}\.

As a composition\-only baseline, we scoren=100n\{=\}100random sequences whose residues are sampled independently and identically distributed \(iid\) fromqAq\_\{\\text\{A\}\}atL=400L\{=\}400on all three metrics \(Table[4](https://arxiv.org/html/2606.27939#S5.T4)\)\. Any policy scoring close to this iid floor would be statistically indistinguishable from a random composition\-matched sequence in terms of solubility and protein\-likeness\. The iid floor on ESM\-2 pPPL is18\.0718\.07\(approximately length\-independent in the relevant range; App\.[G](https://arxiv.org/html/2606.27939#A7)\), the iid solubility floor is0\.492±0\.0550\.492\\pm 0\.055, and the iid*base log\-PPL*floor is9\.22±0\.249\.22\\pm 0\.24\. Three observations follow\. First, predicted solubility on RL policies \(0\.5820\.582onqAq\_\{\\text\{A\}\}\) sits about1\.61\.6floor\-std above the iid floor \(0\.4920\.492\)\. The gap is modest but consistent in sign, so policy solubility is not entirely a composition artefact\. Second, the RL aggregate onqAq\_\{\\text\{A\}\}under*base log\-PPL*\(9\.08±1\.139\.08\\pm 1\.13\) is just below the iid floor \(9\.22±0\.249\.22\\pm 0\.24\); per\-variant,*differentiated*\(8\.828\.82\) and*cosine*\(7\.357\.35\) sit clearly below the floor, while*global\-deviation*\(9\.989\.98\) and*symmetric*\(10\.1810\.18\) cross above it\. Third, all RL policies remain measurably below the ESM\-2 pPPL floor onqAq\_\{\\text\{A\}\}, so even the worst variant has not collapsed to ”random AA” plausibility\. The ESM\-2 pPPL gap is, however, variant\-dependent: the*differentiated*term \(9\.419\.41\) sits roughly midway between the FT prior and the iid floor \(about1\.6×1\.6\\timesthe FT pPPL and about half the floor\), whereas the*symmetric*term \(16\.6516\.65\) and*global\-deviation*\(15\.7915\.79\) sit close to the floor \(within∼8\\sim 8\-13%13\\%of18\.0718\.07\) \(Table[3](https://arxiv.org/html/2606.27939#S5.T3)\)\. This quantifies the qualitative claim above, thatLpL\_\{p\}\- and*symmetric*composition terms approach the regime of pure\-composition random sequences under ESM\-2, while the biologically weighted*differentiated*term retains a clear plausibility margin over that floor\.

As a diagnostic on the KL anchor of Eq\.[1](https://arxiv.org/html/2606.27939#S3.E1), we report the*reference log\-PPL*\(policy mean log\-perplexity under its own FT prior\), a one\-sided proxy for the KL term \(differs fromKL​\(πθ∥πref\)\\mathrm\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)by the policy entropy\)\. The RL aggregate is8\.30±1\.358\.30\\pm 1\.35onqAq\_\{\\text\{A\}\}and4\.83±0\.724\.83\\pm 0\.72onqBq\_\{\\text\{B\}\}, at or below the iid references under the corresponding FT priors \(8\.52±0\.218\.52\\pm 0\.21and8\.90±0\.178\.90\\pm 0\.17; App\.[D](https://arxiv.org/html/2606.27939#A4)\), so no runaway drift on either target\. The larger margin onqBq\_\{\\text\{B\}\}is consistent with it being the easier target \(no zero\-target residues\)\. The overlap with the iid reference is expected rather than informative about plausibility, since the FT prior was trained on composition\-filtered data, that the same iid\-qAq\_\{\\text\{A\}\}sequences score9\.22±0\.249\.22\\pm 0\.24under base ProtGPT2 \(Table[4](https://arxiv.org/html/2606.27939#S5.T4)\) vs\.8\.528\.52under the FT model, confirming the FT prior has internalized the composition and is a meaningful KL anchor\. Target\-independent plausibility is read from base log\-PPL and ESM\-2 pPPL\.

Finally, we verify that the RL stage preserves sequence diversity\. Table[5](https://arxiv.org/html/2606.27939#S5.T5)reports two complementary intra\-pool similarity metrics aggregated over all RL runs \(lower==more diverse\)\. First,*Aln\. id\.*, the mean MMseqs2 identity over alignment\-survivors at coverage≥0\.6\\geq 0\.6, which is alignment\-conditional and therefore upward\-biased toward the most\-similar pair tail, and second,*44\-mer Jacc\.*, the mean pairwise Jaccard of44\-mer sets on a fixed200200\-sequence subsample, which is defined for every pair and so reads the whole pool\. Two anchors interpret the magnitudes \(Table[5](https://arxiv.org/html/2606.27939#S5.T5),qA/qBq\_\{\\text\{A\}\}\\,/\\,q\_\{\\text\{B\}\}stacked per cell\)\. Pre\-RL pools of500500sequences from the FT prior reach*44\-mer Jacc*\.∼0\.01\\sim 0\.01on both targets, which we read as the unconditioned\-generation floor atL∼300L\\\!\\sim\\\!300AAs\. The RL aggregate \(0\.2460\.246onqAq\_\{\\text\{A\}\},0\.4830\.483onqBq\_\{\\text\{B\}\}\) is measurably more concentrated than this floor, as expected for a policy optimized toward a fixed target composition, but stays well clear of the collapsed RL\-only seed of §[5\.5\.2](https://arxiv.org/html/2606.27939#S5.SS5.SSS2)\(0\.740\.74*44\-mer Jacc\.*,0\.970\.97*Aln\. id\.*\)\. Per\-variant breakdown is in App\.[D](https://arxiv.org/html/2606.27939#A4)\(Table[11](https://arxiv.org/html/2606.27939#A4.T11)\), and the diversity\-pulse mechanism that maintains the margin during training is detailed in App\.[H](https://arxiv.org/html/2606.27939#A8)\(Fig\.[5](https://arxiv.org/html/2606.27939#A8.F5)\)\. Mean sequence length stays inside the target interval\[70,400\]\[70,400\]AA\.

### 5\.5Ablation Studies

We report three controlled ablations: the contribution of the*differentiated*composition term’s design choices relative to ablated and baseline alternatives \(*symmetric*,*cosine*, and*global\-deviation*\) onqAq\_\{\\text\{A\}\}\(§[5\.5\.1](https://arxiv.org/html/2606.27939#S5.SS5.SSS1)\), the necessity of the FT stage onqAq\_\{\\text\{A\}\}\(§[5\.5\.2](https://arxiv.org/html/2606.27939#S5.SS5.SSS2)\), and best\-of\-N selection from the FT prior on both targets \(§[5\.5\.3](https://arxiv.org/html/2606.27939#S5.SS5.SSS3)\)\.

Table 5:Intra\-pool sequence similarity \(lower==more diverse\), aggregated over all RL runs per target \(n=120n\{=\}120onqAq\_\{\\text\{A\}\},n=40n\{=\}40onqBq\_\{\\text\{B\}\}; mean±\\pmstd\), and on two pre\-RL anchor pools of500500sequences each \(base ProtGPT2 and the domain\-adaptive FT prior,qA/qBq\_\{\\text\{A\}\}\\,/\\,q\_\{\\text\{B\}\}stacked per cell\)\.*Aln\. id\.*is mean MMseqs2 identity over alignment\-survivors at coverage≥0\.6\\geq 0\.6\(alignment\-conditional, upward\-biased; for the pre\-RL anchors, the average is over only2828\-610610surviving pairs because most pairs do not align at this threshold, so cross\-row comparison should rely on44\-mer Jacc\.\)\.*44\-mer Jacc\.*is the mean pairwise Jaccard of44\-mer sets on a fixed200200\-sequence subsample \(defined for every pair, hence directly comparable across rows\)\.Table 6:FT→\\rightarrowRL vs RL\-only onqAq\_\{\\text\{A\}\}, mean acrossn=3n\{=\}3paired seeds \(same recipe and budget\)\. Comp\. is the composition score atβref=20\\beta\_\{\\text\{ref\}\}\{=\}20; Ess\. and Pools are sequence constraint indicators \(§[4](https://arxiv.org/html/2606.27939#S4)\)\.Table 7:Reward\-formulation comparison onqAq\_\{\\text\{A\}\}\(mean±\\pmstd acrossn=30n=30seeds, matched compute\)\. Solub\. is mean NetSolP predicted solubility \(higher==more soluble\)\. Best per column in bold\.#### 5\.5\.1Composition term formulation

We evaluate the design choices of the*differentiated*composition term against one ablated \(*symmetric*\) and two baseline \(*cosine*and*global\-deviation*\) composition terms onqAq\_\{\\text\{A\}\}at matched compute and shared seeds \(Table[7](https://arxiv.org/html/2606.27939#S5.T7)\)\. Pairwise Wilcoxon signed\-rank tests on then=30n\{=\}30paired seeds give a clear ranking: every smooth\-max \(*differentiated*,*symmetric*\) or*cosine*term beats theL1L\_\{1\}term \(*global\-deviation*\) on composition score \(all surviving Bonferroni correction atα=0\.05\\alpha=0\.05\), and on JSD for*symmetric*over*global\-deviation*\. The remaining three variants \(differentiated, symmetric, cosine\) are not pairwise separable on JSD, and only*differentiated*vs\.*cosine*separates uncorrected on the composition score\. Full pairwise statistics are in App\.[D\.1](https://arxiv.org/html/2606.27939#A4.SS1); per\-variant seed distributions in App\.[D](https://arxiv.org/html/2606.27939#A4)\.

On this target, the statistical comparison confirms that any smooth\-max\- or cosine\-based composition term substantially outperforms theL1L\_\{1\}baseline, while the three upper\-cluster variants are not reliably separable on aggregate metrics alone\. However, the*differentiated*term is our default choice for two reasons that go beyond aggregate composition metrics alone\. First, it directly encodes biological constraints, penalizing deficits in essential AA more harshly than excesses, handling interchangeable AA pools as group\-level terms, and amplifying deviations on zero\-target AA, which are design choices motivated by the nutritional objective rather than arbitrary hyperparameters\. Second, it incurs the smallest pPPL penalty among the smooth\-max variants \(§[5\.4](https://arxiv.org/html/2606.27939#S5.SS4), Table[3](https://arxiv.org/html/2606.27939#S5.T3)\), suggesting that biological weighting partially mitigates the plausibility cost of composition alignment\.

#### 5\.5\.2Domain\-adaptive FT vs RL\-only

We assess the necessity of the domain\-adaptive FT stage by running RL directly on base ProtGPT2 \(RL\-only\), onn=3n\{=\}3seeds shared with the FT→\\toRL runs \(Table[6](https://arxiv.org/html/2606.27939#S5.T6)\)\. Mean paired differences \(FT→\\toRL minus RL\-only across the three seeds\) favor FT→\\toRL on every metric: composition\+0\.15\+0\.15, essential\-residue coverage\+0\.40\+0\.40, pool tolerance\+0\.41\+0\.41,N±30\+1\.4N\_\{\\pm 30\}\{\+\}1\.4, and JSD−0\.014\-0\.014\(lower is better\)\. Given onlyn=3n\{=\}3paired seeds, we report direction and magnitude without formal significance testing\. The FT→\\toRL values of this subsection are restricted to the three seeds matched to the RL\-only version, and it is therefore not directly comparable to then=30n\{=\}30mean reported in §[5\.2](https://arxiv.org/html/2606.27939#S5.SS2)\. Per\-seed inspection reveals a characteristic reward\-hacking failure mode that emerges when RL is run without the FT prior, that is, one of the three RL\-only seeds collapsed onto a very narrow AA palette, satisfying low\-target compliance perfectly \(no zero\-target AA ever exceeded its cap\), but in doing so, it dropped essential\-residue coverage to zero\. Despite this, it still scored a high composition score of0\.700\.70because the average frequency vector remained close toqAq\_\{\\text\{A\}\}\.

The collapse is also visible as a diversity signature\. The failed RL\-only seed reaches a mean intra\-pool pairwise identity of0\.970\.97\(near saturation;*Aln\. id\.*\) and*44\-mer Jaccard*of0\.740\.74, versus0\.840\.84/0\.890\.89*Aln\. id\.*and0\.210\.21/0\.310\.31*44\-mer Jacc*\. for the surviving RL\-only seeds, and0\.720\.72\-0\.820\.82*Aln\. id\.*\(0\.520\.52\-0\.570\.5744\-mer Jacc\.\) for the matched FT→\\toRL seeds, suggesting on both metrics that, without the FT prior, the policy contracts onto a narrow palette of nearly\-identical sequences\. This single seed inflates the RL\-only mean composition score in Table[6](https://arxiv.org/html/2606.27939#S5.T6), but the other two RL\-only seeds average composition≈0\.29\\approx 0\.29\. FT prevents this failure mode in our sweep, though we note this is based on a single RL\-only failure case and a limitedn=3n=3paired comparison\. We read the mechanism as a reward\-landscape effect rather than a property of the FT prior in isolation\. The FT prior is itself narrower than base ProtGPT2 \(its training set is composition\-filtered\), but it sits much closer toqq\(JSD​0\.059\\mathrm\{JSD\}\\,0\.059vs\.0\.2470\.247\), so reward\-weighted updates are spread across many moderately\-rewarding directions and the KL penalty againstπref\\pi\_\{\\text\{ref\}\}\(§[3\.3](https://arxiv.org/html/2606.27939#S3.SS3)\) actively pulls the policy back toward this broad composition\-conditioned region\. From base ProtGPT2, the same KL budget cannot reach a comparably\-rewarded region without latching onto a few high\-reward modes, which is the collapse signature observed here\.

#### 5\.5\.3Best\-of\-N from the FT prior

To test whether reward\-weighted RL updates merely reproduce best\-of\-N selection from the FT prior, we draw500500sequences from the FT\-only model and from base ProtGPT2, and rank them by the same fixed\-βref\\beta\_\{\\text\{ref\}\}composition score used throughout\. OnqAq\_\{\\text\{A\}\}, the best\-of\-500500from FT\-only reachesJSD=0\.018\\mathrm\{JSD\}=0\.018\(composition score=0\.529=0\.529\), while the best FT→\\toRL run reachesJSD=0\.0044\\mathrm\{JSD\}=0\.0044\(composition score=0\.822=0\.822\)\. This is a4×4\\timesreduction in JSD that no amount of selection from the FT prior recovers in500500draws\. OnqBq\_\{\\text\{B\}\}, best\-of\-500500from FT\-only reachesJSD=0\.0111\\mathrm\{JSD\}=0\.0111vs\.0\.00080\.0008for the best FT→\\toRL run, a14×14\\timesgap\. Base ProtGPT2 is two orders of magnitude further from either target even at the best\-of\-500500tail \(Table[14](https://arxiv.org/html/2606.27939#A5.T14)in App\.[E](https://arxiv.org/html/2606.27939#A5)\)\. The RL stage, therefore, moves the policy into a region of sequence space that the FT prior does not reach by oversampling alone\.

## 6Discussion

This work demonstrates that steering a PLM toward an explicit distributional target AA composition benefits from two training stages addressing different aspects of the problem\. Domain\-adaptive FT shifts the base model toward an average composition close toqq, but cannot enforce the sequence constraints needed to select individual candidates for synthesis\. This per\-residue gap is largest on residues whose target frequency is far from the natural protein average\. The RL stage closes this gap, taking the FT prior from weak sequence constraint satisfaction to near\-saturation under the best aligned policy, a transition that oversampling from the FT prior alone cannot reproduce\. Removing the FT stage degrades every metric on every paired seed, and one of the three RL\-only seeds exhibits a discrete failure mode in which the policy reward\-hacks by collapsing onto a narrow residue palette that satisfies the target composition on average while suppressing essential residue coverage\. This failure mode is absent from all FT→\\toRL runs in our sweep, supporting the view that the domain\-adaptive FT provides a stable initialization that the reward signal alone cannot guarantee\.

A key contribution of this work is the*differentiated*composition term, a novel reward formulation tailored to the specific task of designing digestible proteins with a nutrition\-oriented AA composition\. To evaluate its design choices, we compare it against an ablated variant \(*symmetric*\) and two existing baselines \(*cosine*,*global\-deviation*\) in a controlled seed\-matched experiment\. The results show that the*global\-deviation*baseline is clearly insufficient while the remaining three terms form a broadly comparable upper cluster\. A weak signal suggests that smooth\-max variants may outperform the*cosine*term, though this does not survive Bonferroni correction at the seed counts used here\. Within the upper cluster, the*differentiated*term attains the highest mean composition score on theqAq\_\{A\}target composition while its best run saturates all sequence constraints \(essential\-residue coverage, pool tolerance, low\-target compliance\), making it the most faithful to the nutritional design objective\. Additionally,*differentiated*carries the smallest ESM\-2 pPPL penalty, indicating that the composition gains are not bought at a disproportionate plausibility cost, and suggesting that biologically grounded reward design may inherently produce more natural\-looking sequences than uniform alternatives\.

Several limitations should be noted\. First, the pipeline does not enforce sequence plausibility beyond the regularization provided by the KL penalty against the FT reference model, and we do not measure structure or function directly\. Second, seed counts are modest \(3030onqAq\_\{\\text\{A\}\}and1010onqBq\_\{\\text\{B\}\}\), so we report bootstrap CIs rather than asymptotic null tests\. Third, biological follow\-up, including digestibility\-related proxies, is necessary to determine whether improved target composition alignment also yields biologically plausible candidates\. Addressing these limitations represents the natural next step for this line of work\.

To the best of our knowledge, this is the first work to steer a PLM toward an explicit target AA composition as a primary design objective\. While matching a frequency vector over 20 AA may appear to be a simple distributional matching problem, it is better understood as a multi\-objective sequence design task: generated sequences must simultaneously satisfy a range length, essential\-residue coverage, interchangeable pool balance, and zero\-target compliance, without drifting away from the manifold of plausible protein sequences\. That the best aligned policies satisfy all these constraints simultaneously while retaining reasonable ESM\-2 plausibility demonstrates that the proposed two\-stage pipeline, domain\-adaptive FT followed by reward\-weighted RL with a biologically grounded composition term, is a viable approach to this class of constrained generative objectives\.

## Impact Statement

This work develops a method for steering PLMs toward an explicit target AA composition\. The motivating application is feed\-protein design, but the pipeline is broadly applicable to any composition\-constrained protein generation task\. All generated sequences are computational proposals that require further validation before wet\-lab synthesis or downstream deployment\. Depending on the application, this includes structure prediction, disorder analysis, digestibility assays for feed proteins, or activity assays for functional proteins\. Potential positive impacts include better\-aligned dietary proteins for animal feed and a reduced environmental cost of feed production\.

## Acknowledgements

This work was supported by the European Innovation Council \(EIC\) Pathfinder Open project SYNFEED, under grant agreement No\. 101186580\. The European Commission’s support for the production of this publication does not constitute endorsement of the contents, which reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein\.

## References

- M\. Cambra\-López, P\. J\. Marín\-García, C\. Lledó, A\. Cerisuelo, and J\. J\. Pascual \(2022\)Biomarkers and de novo protein design can improve precise amino acid nutrition in broilers\.Animals12\(7\),pp\. 935\.External Links:[Document](https://dx.doi.org/10.3390/ani12070935)Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p2.1),[§1](https://arxiv.org/html/2606.27939#S1.p3.1),[§4\.1](https://arxiv.org/html/2606.27939#S4.SS1.p2.1)\.
- H\. Cao, M\. D\. T\. Torres, J\. Zhang, Z\. Gao, F\. Wu, C\. Gu, J\. Leskovec, Y\. Choi, C\. de la Fuente\-Nunez, G\. Chen, and P\. Heng \(2025\)A deep reinforcement learning platform for antibiotic discovery\.bioRxiv\.Note:PreprintExternal Links:[Document](https://dx.doi.org/10.1101/2025.09.23.678086)Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1),[§2](https://arxiv.org/html/2606.27939#S2.p1.2)\.
- J\. L\. Emmert and D\. H\. Baker \(1997\)Use of the ideal protein concept for precision formulation of amino acid levels in broiler diets\.Journal of Applied Poultry Research6\(4\),pp\. 462–470\.External Links:[Document](https://dx.doi.org/10.1093/japr/6.4.462)Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p2.1)\.
- N\. Ferruz, S\. Schmidt, and B\. Höcker \(2022\)ProtGPT2 is a deep unsupervised language model for protein design\.Nature Communications13,pp\. 4348\.Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1),[§3\.2](https://arxiv.org/html/2606.27939#S3.SS2.p2.1)\.
- S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. Smith \(2020\)Don’t stop pretraining: adapt language models to domains and tasks\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL 2020\),pp\. 8342–8360\.Cited by:[§3\.2](https://arxiv.org/html/2606.27939#S3.SS2.p1.18)\.
- D\. Hesslow, N\. Zanichelli, P\. Notin, I\. Poli, and D\. Marks \(2022\)RITA: a study on scaling up generative protein sequence models\.arXiv preprint arXiv:2205\.05789\.Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1)\.
- Z\. Lin, H\. Akin, R\. Rao, B\. Hie, Z\. Zhu, W\. Lu, N\. Smetanin, R\. Verkuil, O\. Kabeli, Y\. Shmueli, A\. dos Santos Costa, M\. Fazel\-Zarandi, T\. Sercu, S\. Candido, and A\. Rives \(2023\)Evolutionary\-scale prediction of atomic\-level protein structure with a language model\.Science379\(6637\),pp\. 1123–1130\.Cited by:[§5\.1](https://arxiv.org/html/2606.27939#S5.SS1.p2.17),[§5\.4](https://arxiv.org/html/2606.27939#S5.SS4.p3.10)\.
- A\. Madani, B\. Krause, E\. R\. Greene, S\. Subramanian, B\. P\. Mohr, J\. M\. Holton, J\. L\. Olmos, C\. Xiong, Z\. Z\. Sun, R\. Socher, J\. S\. Fraser, and N\. Naik \(2023\)Large language models generate functional protein sequences across diverse families\.Nature Biotechnology41,pp\. 1099–1106\.Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1)\.
- G\. Munsamy, S\. Lindner, P\. Lorenz, and N\. Ferruz \(2024\)Conditional language models enable the efficient design of proficient enzymes\.bioRxiv\.Note:PreprintExternal Links:[Document](https://dx.doi.org/10.1101/2024.05.03.592223)Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1)\.
- E\. Nijkamp, J\. A\. Ruffolo, E\. N\. Weinstein, N\. Naik, and A\. Madani \(2023\)ProGen2: exploring the boundaries of protein language models\.Cell Systems14\(11\),pp\. 968–978\.e3\.External Links:[Document](https://dx.doi.org/10.1016/j.cels.2023.10.002)Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1)\.
- X\. B\. Peng, A\. Kumar, G\. Zhang, and S\. Levine \(2019\)Advantage\-weighted regression: simple and scalable off\-policy reinforcement learning\.arXiv preprint arXiv:1910\.00177\.Cited by:[§2](https://arxiv.org/html/2606.27939#S2.p2.1),[§3\.3](https://arxiv.org/html/2606.27939#S3.SS3.p1.12)\.
- J\. Peters and S\. Schaal \(2007\)Reinforcement learning by reward\-weighted regression for operational space control\.InProceedings of the 24th International Conference on Machine Learning \(ICML 2007\),pp\. 745–750\.Cited by:[§2](https://arxiv.org/html/2606.27939#S2.p2.1),[§3\.3](https://arxiv.org/html/2606.27939#S3.SS3.p1.12)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),Cited by:[§2](https://arxiv.org/html/2606.27939#S2.p1.2),[§2](https://arxiv.org/html/2606.27939#S2.p2.1),[§3\.3](https://arxiv.org/html/2606.27939#S3.SS3.p1.12)\.
- V\. Ravindran \(2013\)Feed enzymes: the science, practice, and metabolic realities\.Journal of Applied Poultry Research22\(3\),pp\. 628–636\.Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p2.1)\.
- M\. Steinegger and J\. Söding \(2017\)MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets\.Nature Biotechnology35\(11\),pp\. 1026–1028\.Cited by:[§3\.3](https://arxiv.org/html/2606.27939#S3.SS3.p2.5)\.
- F\. Stocco, M\. Artigues\-Lleixà, A\. Hunklinger, T\. Widatalla, M\. Güell, and N\. Ferruz \(2024\)Guiding generative protein language models with reinforcement learning\.arXiv preprint arXiv:2412\.12979\.Note:PreprintCited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1),[§2](https://arxiv.org/html/2606.27939#S2.p1.2)\.
- J\. Subramanian, S\. Sujit, N\. Irtisam, U\. Sain, R\. Islam, D\. Nowrouzezahrai, and S\. Ebrahimi Kahou \(2024\)Reinforcement learning for sequence design leveraging protein language models\.arXiv preprint arXiv:2407\.03154\.Note:PreprintCited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1),[§2](https://arxiv.org/html/2606.27939#S2.p1.2)\.
- The UniProt Consortium \(2023\)UniProt: the Universal Protein Knowledgebase in 2023\.Nucleic Acids Research51\(D1\),pp\. D523–D531\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkac1052)Cited by:[Appendix A](https://arxiv.org/html/2606.27939#A1.p2.9),[§3\.2](https://arxiv.org/html/2606.27939#S3.SS2.p1.18)\.
- V\. Thumuluri, J\. J\. Almagro Armenteros, A\. R\. Johansen, H\. Nielsen, and O\. Winther \(2022\)NetSolP: predicting protein solubility in*Escherichia coli*using language models\.Bioinformatics38\(4\),pp\. 941–946\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btab801)Cited by:[§4\.3](https://arxiv.org/html/2606.27939#S4.SS3.p3.1)\.
- T\. Widatalla, R\. Rafailov, and B\. Hie \(2024\)Aligning protein generative models with experimental fitness via direct preference optimization\.bioRxiv\.Note:PreprintExternal Links:[Document](https://dx.doi.org/10.1101/2024.05.20.595026)Cited by:[§1](https://arxiv.org/html/2606.27939#S1.p1.1),[§2](https://arxiv.org/html/2606.27939#S2.p1.2)\.

## Appendix AReproducibility details

This appendix consolidates the practical settings used to produce every number in the main text\. All runs used a single NVIDIA H100 GPU and a single shared conda environment\.

Dataset construction\.Filters and final dataset sizes \(§[3\.2](https://arxiv.org/html/2606.27939#S3.SS2)\): UniProtKB/TrEMBL\(The UniProt Consortium,[2023](https://arxiv.org/html/2606.27939#bib.bib21)\)\(FASTA from the UniProt Consortium FTP\)→\\tolength filter\[100,500\]\[100,500\]aa→\\toper\-sequence cosine similarity toqq≥0\.95\\geq 0\.95→\\toMMseqs2 redundancy at<70%<70\\%pairwise identity, yielding∼\\sim1\.0×1051\.0\{\\times\}10^\{5\}sequences split 50/50 into train and eval per target\.

Stage 1 \(FT\)\.ProtGPT2 base; ProtGPT2 tokenizer; block size512512; AdamW with learning rate5×10−55\{\\times\}10^\{\-5\}, weight decay0\.010\.01,3%3\\%linear warm\-up; effective batch size3232blocks; up to4040epochs with early stopping on eval loss \(patience77,min⁡Δ=10−3\\min\\Delta=10^\{\-3\}\)\. The selected checkpoint forqAq\_\{\\text\{A\}\}is at epoch2121; total wall\-clock∼\\sim13\.513\.5h\. The same checkpoint is reused asπref\\pi\_\{\\text\{ref\}\}for every RL run onqAq\_\{\\text\{A\}\}whileqBq\_\{\\text\{B\}\}uses its own FT checkpoint trained with identical hyperparameters on theqBq\_\{\\text\{B\}\}filtered dataset\.

Stage 2 \(RL\)\.Settings shared across all reported runs \(Table[8](https://arxiv.org/html/2606.27939#A1.T8)\)\.

Table 8:Stage\-2 \(RL\) hyperparameters\. Values are fixed across all160160runs reported in this paper\.Run breakdown\.The full4×30\+4×10=1604\{\\times\}30\+4\{\\times\}10=160runs of §[4\.2](https://arxiv.org/html/2606.27939#S4.SS2)share the loop and optimizer settings of Table[8](https://arxiv.org/html/2606.27939#A1.T8)\. The only differences between runs are the random seed, the composition variant, and the target\.

Evaluation protocol\.For each run, we select the iteration that maximizes the fixedβref=20\\beta\_\{\\text\{ref\}\}\{=\}20composition score on its candidate pool, and report all metrics on that pool\. Per\-variant confidence intervals are seed\-level95%95\\%percentile bootstraps with10410^\{4\}resamples\.

## Appendix BComposition term formulas

We give explicit formulas for all composition terms used at training and evaluation time\. Letp∈𝒮20p\\in\\mathcal\{S\}\_\{20\}be the observed AA frequency vector of a candidate sequence \(over the 20 standard AA\) andq∈𝒮20q\\in\\mathcal\{S\}\_\{20\}the target frequency vector\. LetLLdenote sequence length and letℰ\\mathcal\{E\}denote the essential\-AA index set used in the*differentiated*composition\.

Per\-residue error \(*differentiated*\)\.For each AAii,

ei=\{w−​\(qi−pi\)i∈ℰ,pi<qi,w\+​\(pi−qi\)i∈ℰ,pi≥qi,wne​\|pi−qi\|i∉ℰ,e\_\{i\}=\\begin\{cases\}w^\{\-\}\(q\_\{i\}\-p\_\{i\}\)&i\\in\\mathcal\{E\},\\ p\_\{i\}<q\_\{i\},\\\\ w^\{\+\}\(p\_\{i\}\-q\_\{i\}\)&i\\in\\mathcal\{E\},\\ p\_\{i\}\\geq q\_\{i\},\\\\ w^\{\\text\{ne\}\}\\,\|p\_\{i\}\-q\_\{i\}\|&i\\notin\\mathcal\{E\},\\end\{cases\}withw−=3\.0w^\{\-\}=3\.0,w\+=0\.35w^\{\+\}=0\.35,wne=1\.0w^\{\\text\{ne\}\}=1\.0, plus a zero\-target amplifierei←α0​eie\_\{i\}\\leftarrow\\alpha\_\{0\}\\,e\_\{i\}wheneverqi=0q\_\{i\}=0andpi\>0p\_\{i\}\>0\(withα0=3\\alpha\_\{0\}=3\)\. Per\-residue errors are clipped to≤2\\leq 2\. Group\-level errorswgrp​\|∑i∈Gpi−tG\|w^\{\\text\{grp\}\}\\,\|\\sum\_\{i\\in G\}p\_\{i\}\-t\_\{G\}\|for interchangeable AA groupsGG\(withwgrp=1\.5w^\{\\text\{grp\}\}=1\.5; onqAq\_\{\\text\{A\}\}these are the sulfur and aromatic\-precursor pools, §[4](https://arxiv.org/html/2606.27939#S4)\) are appended to the error vectoree\.

*Differentiated*score\.Withα=60\\alpha=60,wrms=1w\_\{\\text\{rms\}\}=1,wsm=0\.9w\_\{\\text\{sm\}\}=0\.9, andβ\\betaramped15→3615\{\\to\}36across training iterations, and lettingrms​\(e\)=1n​∑iei2\\mathrm\{rms\}\(e\)=\\sqrt\{\\tfrac\{1\}\{n\}\\sum\_\{i\}e\_\{i\}^\{2\}\}andsmaxα​\(e\)=1α​log⁡\(1n​∑ieα​ei\)\\mathrm\{smax\}\_\{\\alpha\}\(e\)=\\tfrac\{1\}\{\\alpha\}\\log\\\!\\big\(\\tfrac\{1\}\{n\}\\sum\_\{i\}e^\{\\alpha e\_\{i\}\}\\big\),

Compdiff​\(p;q\)=exp⁡\(−β​\[wrms​rms​\(e\)\+wsm​smaxα​\(e\)\]\)\.\\mathrm\{Comp\}\_\{\\text\{diff\}\}\(p;q\)=\\exp\\\!\\Big\(\\\!\-\\beta\\big\[w\_\{\\text\{rms\}\}\\,\\mathrm\{rms\}\(e\)\+w\_\{\\text\{sm\}\}\\,\\mathrm\{smax\}\_\{\\alpha\}\(e\)\\big\]\\Big\)\.The second bracket term is a smooth\-max \(log\-sum\-exp\) over the per\-residue errors with sharpnessα\\alpha\. Evaluation usesβref=20\\beta\_\{\\text\{ref\}\}=20\.

*Symmetric*score\.Same RMS \+ smooth\-max kernel, but with uniform absolute deviationei=\|pi−qi\|e\_\{i\}=\|p\_\{i\}\-q\_\{i\}\|\(no essential split, no group terms\)\. The remaining hyperparameters\(α,wrms,wsm,α0\)\(\\alpha,w\_\{\\text\{rms\}\},w\_\{\\text\{sm\}\},\\alpha\_\{0\}\)are kept identical to the*differentiated*variant; only the outer sharpnessβ\\betais ramped on a different schedule,30→8030\{\\to\}80across training iterations, to match the loss scale of the simpler kernel:

Compsym​\(p;q\)=exp⁡\(−β​\[wrms​rms​\(e\)\+wsm​smaxα​\(e\)\]\)\.\\mathrm\{Comp\}\_\{\\text\{sym\}\}\(p;q\)=\\exp\\\!\\Big\(\\\!\-\\beta\\big\[w\_\{\\text\{rms\}\}\\,\\mathrm\{rms\}\(e\)\+w\_\{\\text\{sm\}\}\\,\\mathrm\{smax\}\_\{\\alpha\}\(e\)\\big\]\\Big\)\.*differentiated*and*symmetric*therefore share the sameRMS\+smooth​\-​max\\mathrm\{RMS\}\+\\mathrm\{smooth\\text\{\-\}max\}kernel; the*differentiated*variant adds biologically\-motivated essential/non\-essential weighting and the interchangeable\-group terms, while*symmetric*uses uniform\|pi−qi\|\|p\_\{i\}\-q\_\{i\}\|throughout\.

*Cosine*score\.

Compcos​\(p;q\)=exp⁡\(−β​\[1−p⋅q‖p‖​‖q‖\]\),\\mathrm\{Comp\}\_\{\\text\{cos\}\}\(p;q\)=\\exp\\\!\\Big\(\-\\beta\\big\[1\-\\tfrac\{p\\cdot q\}\{\\\|p\\\|\\,\\\|q\\\|\}\\big\]\\Big\),withβ\\betaramped5→255\{\\to\}25across training iterations\. Scale\-invariant: matches direction inℝ20\\mathbb\{R\}^\{20\}, not magnitude, which is why it lags on theL1L\_\{1\}metric\.

*Global\-deviation*score\.

Compgd​\(p;q\)=exp⁡\(−β​120​∑i\|pi−qi\|\),\\mathrm\{Comp\}\_\{\\text\{gd\}\}\(p;q\)=\\exp\\\!\\Big\(\-\\beta\\,\\tfrac\{1\}\{20\}\\textstyle\\sum\_\{i\}\|p\_\{i\}\-q\_\{i\}\|\\Big\),withβ\\betaramped20→10020\{\\to\}100across iterations\. The simplest baseline: mean\-L1L\_\{1\}deviation, no smooth\-max, no essential split\.

Per\-variantβ\\betaramps\.The four ramps above \(15→3615\{\\to\}36,30→8030\{\\to\}80,5→255\{\\to\}25,20→10020\{\\to\}100\) were chosen based on the best results that were achieved\. Within each variant,β\\betais linearly ramped over a fixed window of iterations \(starting at iteration99, target value reached by iteration3333, then held constant\)\.

Reference\-β\\betare\-scoring\.For all aggregate comparisons, we rescore each candidate pool with the*differentiated*formula atβref=20\\beta\_\{\\text\{ref\}\}\{=\}20, regardless of which variant was used during training\. This puts every variant on a single common scoring function and removes confounding from the per\-variantβ\\betaschedule\.

Length term\.A piecewise\-linear shaping term over the support interval\[Lmin,Lmax\]\[L\_\{\\text\{min\}\},L\_\{\\text\{max\}\}\]with peak plateau\[La,Lb\]\[L\_\{a\},L\_\{b\}\]:

Rℓ​\(L\)=\{0L≤Lmin​or​L≥Lmax,L−LminLa−LminLmin<L<La,1La≤L≤Lb,Lmax−LLmax−LbLb<L<Lmax,R\_\{\\ell\}\(L\)=\\begin\{cases\}0&L\\leq L\_\{\\text\{min\}\}\\ \\text\{or\}\\ L\\geq L\_\{\\text\{max\}\},\\\\ \\dfrac\{L\-L\_\{\\text\{min\}\}\}\{L\_\{a\}\-L\_\{\\text\{min\}\}\}&L\_\{\\text\{min\}\}<L<L\_\{a\},\\\\\[2\.0pt\] 1&L\_\{a\}\\leq L\\leq L\_\{b\},\\\\\[2\.0pt\] \\dfrac\{L\_\{\\text\{max\}\}\-L\}\{L\_\{\\text\{max\}\}\-L\_\{b\}\}&L\_\{b\}<L<L\_\{\\text\{max\}\},\\end\{cases\}withLmin=70L\_\{\\text\{min\}\}=70,La=110L\_\{a\}=110,Lb=250L\_\{b\}=250,Lmax=400L\_\{\\text\{max\}\}=400AA\. The non\-zero support\[70,400\]\[70,400\]AA is intentionally wider than the length filter\[100,500\]\[100,500\]AA \(§[3\.2](https://arxiv.org/html/2606.27939#S3.SS2)\) so that on\-target sequences just outside the dataset window still receive partial credit\.

## Appendix CTop\-5 runs

Table[9](https://arxiv.org/html/2606.27939#A3.T9)lists the top RL runs onqAq\_\{\\text\{A\}\}across alln=120n\{=\}120runs, with rank columns for both JSD \(lower better\) and fixed\-βref\\beta\_\{\\text\{ref\}\}composition score \(higher better\) so that disagreements between the two rankings are visible\.

Table 9:Top\-5 RL runs onqAq\_\{\\text\{A\}\}\(across alln=120n\{=\}120runs\) ranked by JSD \(lower is better\)\.
## Appendix DComposition\-variant breakdown

This appendix collects the per\-variant figures referenced from §[5\.5\.1](https://arxiv.org/html/2606.27939#S5.SS5.SSS1), and the analogous composition\-variant comparison onqBq\_\{\\text\{B\}\}\(n=10n\{=\}10seeds per variant; Table[10](https://arxiv.org/html/2606.27939#A4.T10)\)\.

Table 10:Reward\-formulation comparison onqBq\_\{\\text\{B\}\}\(mean±\\pmstd acrossn=10n\{=\}10seeds, matched compute\)\. Same metrics, scoring, and selection rule as Table[7](https://arxiv.org/html/2606.27939#S5.T7)\. Best per column in bold\.![Refer to caption](https://arxiv.org/html/2606.27939v1/x3.png)

Figure 3:Per\-variant means with seed\-level95%95\\%bootstrap confidence intervals on JSD, composition score \(re\-scored at fixedβref=20\\beta\_\{\\text\{ref\}\}\{=\}20\), and pool tolerance \(qAq\_\{\\text\{A\}\},n=30n\{=\}30seeds per variant\)\. Dotted line: base ProtGPT2; dashed line: domain\-adaptive FT \(no RL\)\.![Refer to caption](https://arxiv.org/html/2606.27939v1/x4.png)

Figure 4:Seed variance per composition term variant on the composition score \(left, atβref=20\\beta\_\{\\text\{ref\}\}\{=\}20\) and the tolerance countN±30N\_\{\\pm 30\}\(right\) onqAq\_\{\\text\{A\}\},n=30n\{=\}30seeds per variant\. Boxes summarize the seed distribution; jittered points show individual seeds\.Table 11:Per\-variant intra\-pool sequence\-similarity breakdown onqAq\_\{\\text\{A\}\}\(n=30n\{=\}30seeds per variant\) andqBq\_\{\\text\{B\}\}\(n=10n\{=\}10seeds per variant\), mean±\\pmstd \(lower==more diverse\)\.*Aln\. id\.*is mean MMseqs2 identity over alignment\-survivors at coverage≥0\.6\\geq 0\.6\.*44\-mer Jacc\.*is the mean pairwise Jaccard over44\-mer sets on a fixed200200\-sequence subsample\.Table 12:Log\-perplexity of each RL policy under its FT prior \(one\-sided proxy forKL​\(πθ∥πref\)\\mathrm\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\), equal to𝔼πθ​\[−log⁡πref\]\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\[\-\\log\\pi\_\{\\text\{ref\}\}\]up to the policy entropy, §[5\.4](https://arxiv.org/html/2606.27939#S5.SS4)\)\. Each cell is mean±\\pmstd across seeds of per\-run pool means\. Values are*not*directly comparable between targets because each target uses a different FT prior\. The*Aggregate*row pools all four variants per target\. The*iid floor*row reports the same metric onn=100n\{=\}100random sequences sampled iid from the corresponding target atL=400L\{=\}400, scored under that target’s FT prior\.### D\.1Pairwise Wilcoxon: full statistics

Table[13](https://arxiv.org/html/2606.27939#A4.T13)reports the median paired differenceΔ\\Delta,95%95\\%percentile\-bootstrap CI, and Wilcoxonpp\-value for every pair and every aggregate metric onqAq\_\{\\text\{A\}\}\.

Table 13:Full pairwise Wilcoxon signed\-rank statistics onqAq\_\{\\text\{A\}\},n=30n\{=\}30paired seeds\. Boldppsurvives Bonferroni atα=0\.05\\alpha\{=\}0\.05within each metric\.

## Appendix EBest\-of\-NNvs\. RL

Table[14](https://arxiv.org/html/2606.27939#A5.T14)expands the rejection\-sampling argument of §[5\.2](https://arxiv.org/html/2606.27939#S5.SS2)\. We score500500samples from each prior \(base ProtGPT2 and FT\-only\) on each target with the fixed\-βref=20\\beta\_\{\\text\{ref\}\}\{=\}20composition score and report the best and top\-10%10\\%JSD alongside the best composition score\. The best\-of\-500500from the FT prior is closer to the target than from base ProtGPT2 by a factor of two to three on JSD, confirming that domain\-adaptive FT is a real prior shift, but it is still4×4\\times\(qAq\_\{\\text\{A\}\}\) to14×14\\times\(qBq\_\{\\text\{B\}\}\) further from the target than the best RL run\. The gap is largest on the per\-sequence composition score, where the FT prior simply does not contain sequences that satisfy the harder per\-residue constraints\.

Table 14:Rejection\-sampling baseline\. For each target, we drawN=500N\{=\}500sequences from base ProtGPT2 and from the FT\-only model and rank them by the same fixed\-βref\\beta\_\{\\text\{ref\}\}composition score used elsewhere\. We report the best and top\-10%10\\%JSD alongside the best composition score, compared against the best RL run on each target\. Even withN=500N\{=\}500draws, the FT prior alone does not reach the JSD or score that the RL stage reaches in a single run, on either target\. Values for the two “RL best” rows are the per\-run pool means at the best iteration \(n=19n\{=\}19,n=39n\{=\}39valid sequences after filtering onqAq\_\{\\text\{A\}\}andqBq\_\{\\text\{B\}\}respectively\)\.
## Appendix FTop\-3030ESM\-2 pPPL

The per\-condition top\-3030slice referenced in §[5\.4](https://arxiv.org/html/2606.27939#S5.SS4)is computed by ranking each policy’s candidate pool \(the same generation pool that backs Table[3](https://arxiv.org/html/2606.27939#S5.T3)\) by the fixed\-βref\\beta\_\{\\text\{ref\}\}composition score and scoring the top3030with ESM\-2 \(150M\) pPPL \(Table[15](https://arxiv.org/html/2606.27939#A6.T15)\)\. The top\-3030sequences are those most aggressive in matchingqqfrom each policy, so this is the worst\-case fluency snapshot \(rather than the average over the whole pool reported in Table[3](https://arxiv.org/html/2606.27939#S5.T3)\)\. Even on the per\-sequence top\-3030slice, no variant collapses to a degenerate\-low\-perplexity regime: all RL policies remain in the same order of magnitude as their respective FT prior, and the differentiated reward stays closest to the FT prior onqAq\_\{\\text\{A\}\}\.

Table 15:ESM\-2 \(150M\) pseudo\-perplexity \(pPPL\) on the top\-3030composition\-scoring sequences from each policy\. Lower is more natural\-protein\-like\.
## Appendix Giid floor: length dependence

Table[16](https://arxiv.org/html/2606.27939#A7.T16)reports ESM\-2 pPPL of the composition\-only iid baseline atL∈\{100,250,400\}L\\in\\\{100,250,400\\\}AA on both targets\. The floor shrinks by less than11pPPL unit betweenL=100L\{=\}100andL=400L\{=\}400on each target, supporting the use of theL=400L\{=\}400value in Table[4](https://arxiv.org/html/2606.27939#S5.T4)as an approximately length\-independent floor for the RL policies \(whose mean lengths fall inside this range, Table[3](https://arxiv.org/html/2606.27939#S5.T3)\)\.

Table 16:ESM\-2 \(150M\) pseudo\-perplexity \(pPPL\) of the composition\-only iid floor at three lengths, per target \(n=100n\{=\}100sequences each, residues sampled iid from the corresponding target AA distribution; mean±\\pmstd\)\.
## Appendix HDiversity pulse and re\-injection

The diversity\-pulse mechanism introduced in §[3\.3](https://arxiv.org/html/2606.27939#S3.SS3)is parameterized as follows\. The baseline candidate pool is sampled at temperatureT=0\.80T\{=\}0\.80andtop​\-​p=0\.90\\mathrm\{top\\text\{\-\}p\}\{=\}0\.90, and the diversity\-pulse pool is sampled atT=0\.88T\{=\}0\.88andtop​\-​p=0\.93\\mathrm\{top\\text\{\-\}p\}\{=\}0\.93\(i\.e\. both knobs slightly hotter\)\. Both pools pass through the same length and MMseqs2 identity filters before merging\. Two fractions schedule the merge across training:

- •Re\-inject fraction\(carry\-over of merged pool to the next iteration’s prompt set\):0\.350\.35for the first20%20\\%of iterations,0\.450\.45during the early\-boost window, and0\.550\.55during the late \(≥80%\\geq 80\\%\) window\.
- •Pulse mix\-in fraction\(share of the diversity\-pulse pool added to the baseline pool inside one iteration\):0\.200\.20,0\.300\.30, and0\.350\.35on the same three windows\.

Hotter decoding alone would degrade reward\. The re\-injection schedule is what turns the diversity pulse into a useful signal, and the schedule is identical across all160160runs reported in the main text\.

Figure[5](https://arxiv.org/html/2606.27939#A8.F5)summarizes the per\-iteration composition score \(re\-scored at fixedβref=20\\beta\_\{\\text\{ref\}\}=20\) of sequences sampled by the two decoders, aggregated across the top66*differentiated*composition seeds onqAq\_\{\\text\{A\}\}\(matching the seed selection used in Fig\.[1](https://arxiv.org/html/2606.27939#S5.F1)\)\. The diversity\-pulse pool tracks the baseline pool throughout training, lagging it by no more than a few hundredths of a composition\-score unit during the middle of training and converging back to it once the policy has contracted around the target\. The pulse, therefore, acts as a controlled exploration channel\. It does not collapse reward relative to the baseline pool, but it keeps a wider per\-sequence distribution available to the reward\-weighted update during the regime where the policy is still moving\.

![Refer to caption](https://arxiv.org/html/2606.27939v1/x5.png)

Figure 5:Per\-iteration composition score \(re\-scored at fixedβref=20\\beta\_\{\\text\{ref\}\}=20with the*differentiated*variant\) of the baseline pool versus the diversity\-pulse pool, aggregated across the top66*differentiated*\-variant seeds onqAq\_\{\\text\{A\}\}\. Lines: per\-iteration median; shading: per\-iteration2525\-75%75\\%inter\-quartile range across all surviving candidate sequences\. The pulse pool tracks the baseline pool throughout training, lagging it slightly during the contraction regime \(roughly the first3030iterations\) and converging back to it afterwards\.