Q-Steer: Action-Value Guidance for Molecular Policy Optimization

# Q-Steer: Action-Value Guidance for Molecular Policy Optimization
Source: [https://arxiv.org/html/2607.26391](https://arxiv.org/html/2607.26391)
Xinyu Wang1Jinbo Bi1Minghu Song2 1Department of Computer Science and Engineering, University of Connecticut 2Institute of Health and Medicine, Hefei Comprehensive National Science Center 1Storrs, CT 06269, USA2Hefei 230601, China

###### Abstract

Oracle\-limited molecular optimization gives reward only after a complete molecule is generated, while each rollout requires many local next\-token decisions\. This delayed\-feedback interface makes molecular policy optimization myopic: an optimizer can learn that a molecule was good without knowing which intermediate actions made it good\. We introduce Q\-Steer, a rollout\-time action\-value steering primitive for molecular language models\. Q\-Steer uses an offline\-trained and frozen prefix\-action value scorer, PAVS\-Q, that estimates the downstream reward of taking a candidate next token under a partial SMILES prefix, then adds a normalized value bonus to sampling logits\. The optimizer update rule and online oracle budget are unchanged; the claim is fixed\-online\-oracle performance, not equal total compute\. On PMO23 with a fixed 10,000\-call online budget, complete factorial studies across two molecular language\-model backbones and four optimizers show that Q\-Steer improves mean valid\-unique score in all eight backbone–optimizer cells, with positive macro mean\-score gains between \+0\.033 and \+0\.049 and 18–20 task wins per cell\. Mechanism controls show that action identity matters: prefix\-broadcast values are nearly neutral, while shuffled action values harm performance\. These results support Q\-Steer as a reusable rollout\-time action\-value wrapper that improves average molecular optimization reward across optimizer families and policy backbones without changing the online oracle budget\.

## 1Introduction

Goal\-directed molecular generation is usually evaluated under a strict oracle budget: the algorithm may query a property oracle only a limited number of times, and each query requires a completed valid molecule\. This makes molecular policy optimization a delayed\-feedback sequence problem\. A SMILES policy must choose one token at a time, but the reward that drives learning arrives only after termination\. Reinforcement learning methods such as REINVENT and PPO can optimize this interface\(Olivecronaet al\.,[2017](https://arxiv.org/html/2607.26391#bib.bib2); Blaschkeet al\.,[2020](https://arxiv.org/html/2607.26391#bib.bib3); Schulmanet al\.,[2017](https://arxiv.org/html/2607.26391#bib.bib1)\), but their rollout policies remain locally myopic: at a prefix, the generator often lacks action\-level information about which next token will lead to a high\-scoring molecule\.

The standard ways to improve this setting are to change the generator, change the optimizer, or spend the oracle budget differently\. We study a more modular alternative: keep the molecular optimizer and online oracle budget fixed, but make each rollout action less myopic\. If a frozen auxiliary model can estimate the future reward of each candidate next token, then the optimizer can receive local lookahead guidance while still updating exactly as before after the completed molecule is scored\.

We propose Q\-Steer, a rollout\-time action\-value steering primitive for molecular policy optimization\. The guide is PAVS\-Q, a prefix\-action value scorer\. It is the action\-resolved counterpart of a PAVS\-style prefix future\-value prior: rather than only estimating whether a partial molecule is promising, it estimates which next token is promising from that prefix\. Given a prefixx<tx\_\{<t\}and a candidate tokena∈𝒱a\\in\\mathcal\{V\}, PAVS\-Q estimates the downstream molecular reward obtained by takingaanext\. During sampling, Q\-Steer shifts the optimizer’s action logits by a normalized value bonus:

ℓ~t​\(a\)=ℓt​\(a\)\+β​Qϕ​\(x<t,a\),\\tilde\{\\ell\}\_\{t\}\(a\)=\\ell\_\{t\}\(a\)\+\\beta Q\_\{\\phi\}\(x\_\{<t\},a\),\(1\)whereℓt​\(a\)\\ell\_\{t\}\(a\)are the original sampling logits,QϕQ\_\{\\phi\}denotes the centered and scaled PAVS\-Q score over valid actions at the current prefix, andβ\\betais a steering strength\. The intervention is deliberately small\. Within each paired comparison, the policy backbone, optimizer update rule, PMO oracle, and 10,000\-call online budget remain fixed; only the rollout distribution receives action\-level future\-value guidance\.

The design deliberately separates the claim from broader architectural or optimizer changes\. Q\-Steer is not a post\-hoc reranker: it acts before each token is sampled\. It is not a new PPO loss or a new REINVENT objective: completed molecules and rewards are returned to the same optimizer update\. It is also not intended as a diversity mechanism\. The supported claim is narrower and testable: an offline value prior can improve the average score of valid unique molecules at the same online oracle budget\. This is not an equal\-total\-compute claim; offline PAVS\-Q training is disclosed separately, and the frozen guide receives no molecules, rewards, or gradients from the evaluated online trajectory\.

We test this claim with complete PMO23 factorial studies across two molecular language\-model backbones and four optimizer families\. Adding Q\-Steer improves mean valid\-unique score in all eight backbone–optimizer cells under the same 10,000\-call online budget, with 18–20 task wins per cell\. Mechanism controls close the loop: prefix\-broadcast values are nearly neutral, shuffled action values harm performance, and beta sweeps show the observed reward\-diversity tradeoff\. The resulting story is not that Q\-Steer universally improves every metric\. Rather, it is a reusable exploitation primitive: it concentrates search using action\-level future values, improving average reward while reducing uniqueness and leaving top\-k discovery mixed\.

![Refer to caption](https://arxiv.org/html/2607.26391v1/figures/qsteer_overview_manual.png)Figure 1:Conceptual overview\. Baseline molecular optimizers receive terminal oracle feedback after a full SMILES string is sampled, leaving prefix actions locally myopic\. Q\-Steer changes only the rollout distribution: the same optimizer proposes logits over the same valid action set, an offline\-trained frozen PAVS\-Q guide scores those candidate actions, and Q\-Steer adds the bounded logit bonus from Equations[8](https://arxiv.org/html/2607.26391#S3.E8)–[9](https://arxiv.org/html/2607.26391#S3.E9)before the completed molecule is scored once by the PMO oracle\.#### Contributions\.

We make three contributions\. First, we formulate Q\-Steer as a rollout\-time action\-value wrapper: a frozen PAVS\-Q guide scores candidate next tokens and tilts sampling logits without changing the optimizer update or online oracle budget\. Second, we evaluate the wrapper in a paired PMO23 factorial study spanning four optimizer families and two backbones, showing consistent average valid\-unique reward gains in all eight backbone–optimizer cells\. Third, we provide mechanism and limitation evidence: action identity matters, shuffled values hurt, stronger guidance trades reward for uniqueness, and upper\-tail discovery is mixed rather than universally improved\.

## 2Related Work

#### Reinforcement learning for molecular generation\.

Molecular RL methods optimize a generator against property oracles, often using SMILES language models as policies\(Olivecronaet al\.,[2017](https://arxiv.org/html/2607.26391#bib.bib2); Blaschkeet al\.,[2020](https://arxiv.org/html/2607.26391#bib.bib3); Gottipatiet al\.,[2020](https://arxiv.org/html/2607.26391#bib.bib5)\)\. REINVENT\-style approaches regularize an agent toward a prior while increasing the likelihood of high\-scoring molecules, while PPO\-style approaches optimize a clipped policy\-gradient objective over generated trajectories\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.26391#bib.bib1)\)\. Augmented hill\-climb and related likelihood\-ratio updates provide simpler exploitation\-oriented alternatives that repeatedly bias generation toward high\-scoring samples\(Blaschkeet al\.,[2020](https://arxiv.org/html/2607.26391#bib.bib3)\)\. These methods are flexible because the oracle can encode similarity, QSAR activity, MPO objectives, or structural constraints\. Their common limitation is delayed feedback: the reward is attached to a completed molecule, not to the local token actions that produced it\. Q\-Steer is designed to address this local decision problem while keeping the optimizer fixed\.

#### Value guidance and credit assignment\.

Value functions are a standard solution for delayed reward in reinforcement learning, but the way they are used matters\. A state value can tell whether a prefix is promising, but it does not distinguish among next\-token actions from that prefix\. A terminal reward model or reranker can score complete molecules, but it does not intervene during sampling\. Q\-Steer uses a prefix\-action value, which is closer to a molecular analogue of action\-value guidance: it estimates which next token is promising under a specific partial molecule\. The prefix\-broadcast and shuffled\-action controls are included to test this distinction directly\.

#### Decoding\-time steering\.

Language\-model steering methods such as PPLM, GeDi, DExperts, and FUDGE modify token probabilities at generation time using auxiliary models or discriminators\(Dathathriet al\.,[2020](https://arxiv.org/html/2607.26391#bib.bib7); Keskaret al\.,[2019](https://arxiv.org/html/2607.26391#bib.bib8); Krauseet al\.,[2021](https://arxiv.org/html/2607.26391#bib.bib9); Liuet al\.,[2021](https://arxiv.org/html/2607.26391#bib.bib10); Yang and Klein,[2021](https://arxiv.org/html/2607.26391#bib.bib11)\)\. Q\-Steer shares the idea of changing a sampling distribution without retraining the base generator, but differs in both target and signal: the guide is a task\-specific molecular prefix\-action value model, and the controlled variable is online PMO reward under a fixed oracle budget rather than text attributes\. Unlike generic attribute steering, the guide ranks candidate molecular actions by predicted downstream PMO value and is tested by paired optimizer runs rather than by unconditional generation quality\. This connection motivates treatingβ\\betaas a guidance\-strength parameter rather than as a new optimizer objective\.

#### Oracle\-budget benchmarks\.

Oracle\-limited molecular optimization benchmarks emphasize fixed budgets, making it important to report not only the final best molecule but also distributional quality and search dynamics\(Brownet al\.,[2019](https://arxiv.org/html/2607.26391#bib.bib4); Gaoet al\.,[2022](https://arxiv.org/html/2607.26391#bib.bib6)\)\. Mean score, top\-10 reward, cumulative top\-10 AUC, and uniqueness measure different aspects of optimization\. A method may improve average reward while reducing diversity or failing to improve early top\-k discovery\. Our evaluation follows this broader view rather than optimizing one headline metric\.

## 3Method

### 3\.1Delayed\-Reward Molecular Optimization

Letπθ\\pi\_\{\\theta\}be a molecular language\-model policy over token sequencesx=\(x1,…,xT\)x=\(x\_\{1\},\\ldots,x\_\{T\}\), where each token is sampled from a finite vocabulary subject to a SMILES validity mask\. A black\-box oracle returns a scalar rewardR​\(x\)R\(x\)only after the full molecule has been generated and validated\. Under an online oracle budgetBB, a molecular optimizer induces a sequence of rollout distributions and receives at mostBBterminal scores\. Its generic objective can be written as

maxθ⁡𝔼x∼πθ​\[R​\(x\)\],\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\\left\[R\(x\)\\right\],\(2\)but the policy must still choose a local actiona=xta=x\_\{t\}at every prefixx<tx\_\{<t\}before observingR​\(x\)R\(x\)\. This mismatch is the credit\-assignment bottleneck targeted by Q\-Steer\. We do not replace the optimizer that updatesπθ\\pi\_\{\\theta\}; instead, we modify only the sampling distribution used to collect complete molecules\.

### 3\.2From PAVS to PAVS\-Q

The useful object for delayed\-feedback generation is a future\-value prior over partial sequences\. A prefix\-level value model, which we refer to as PAVS, estimates

Vϕ​\(x<t\)≈𝔼​\[R​\(x\)∣x<t\]\.V\_\{\\phi\}\(x\_\{<t\}\)\\approx\\mathbb\{E\}\\left\[R\(x\)\\mid x\_\{<t\}\\right\]\.\(3\)This is informative for deciding whether a partial molecule is promising, but it does not by itself tell the generator which next token to choose\. If the same scalarVϕ​\(x<t\)V\_\{\\phi\}\(x\_\{<t\}\)is added to every valid action logit, the relative probabilities of actions at that prefix are unchanged\.

PAVS\-Q is the action\-resolved version of this prior\. It estimates the value of taking a specific candidate action under a prefix:

Qϕ​\(x<t,a\)≈𝔼​\[R​\(x\)∣x<t,xt=a\]\.Q\_\{\\phi\}\(x\_\{<t\},a\)\\approx\\mathbb\{E\}\\left\[R\(x\)\\mid x\_\{<t\},x\_\{t\}=a\\right\]\.\(4\)Thus PAVS provides the future\-value framing, PAVS\-Q provides token\-level action discrimination, and Q\-Steer is the rollout\-time wrapper that uses these action values to steer sampling\. The distinction is important:V​\(x<t\)V\(x\_\{<t\}\)ranks prefixes, whileQ​\(x<t,a\)Q\(x\_\{<t\},a\)ranks the next actions available from the same prefix\.

### 3\.3Training the Prefix\-Action Value Model

PAVS\-Q is trained offline from scored molecular rollouts\. Given a scored moleculex=\(x1,…,xT\)x=\(x\_\{1\},\\ldots,x\_\{T\}\)with terminal rewardR​\(x\)R\(x\), we create supervised prefix\-action examples

𝒟Q=\{\(x<t,xt,y​\(x\)\):t=1,…,T\},\\mathcal\{D\}\_\{Q\}=\\left\\\{\\left\(x\_\{<t\},x\_\{t\},y\(x\)\\right\):t=1,\\ldots,T\\right\\\},\(5\)wherey​\(x\)y\(x\)is the standardized terminal reward for the PMO task\. The value model is trained by mean\-squared error,

minϕ⁡𝔼\(x<t,a,y\)∼𝒟Q​\(Qϕ​\(x<t,a\)−y\)2\.\\min\_\{\\phi\}\\;\\mathbb\{E\}\_\{\(x\_\{<t\},a,y\)\\sim\\mathcal\{D\}\_\{Q\}\}\\left\(Q\_\{\\phi\}\(x\_\{<t\},a\)\-y\\right\)^\{2\}\.\(6\)In our implementation, PAVS\-Q uses a one\-layer GRU prefix encoder, an action embedding, and an MLP head\. The checkpoint is selected by held\-out prefix\-action validation loss and then frozen before online optimization begins\. The tokenizer and action vocabulary are aligned with the online ACEGEN policy, so the action ID scored by PAVS\-Q is the same action ID sampled by the optimizer\. This alignment is necessary: a value model over a different tokenization would not define a valid action\-level logit bonus\.

For multi\-component PMO objectives, PAVS\-Q can estimate the scalar reward after the task\-specific objective has been applied, or estimate reward components that are combined according to the PMO objective\. The experiments use task\-specific frozen checkpoints\. This offline value\-model training is an additional prior\-building cost, analogous to using a pretrained molecular policy prior\. It is not counted as part of the 10,000\-call online PMO budget, and no generated molecules from the evaluated online trajectory are used to update PAVS\-Q\.

### 3\.4Q\-Steer: Rollout\-Time Action\-Value Steering

At prefixx<tx\_\{<t\}, the base optimizer produces logitsℓt​\(a\)\\ell\_\{t\}\(a\)over valid next\-token actionsa∈𝒜​\(x<t\)a\\in\\mathcal\{A\}\(x\_\{<t\}\)\. Q\-Steer evaluates PAVS\-Q on the same candidate action set and converts raw values into a local normalized bonus,

Q¯t​\(a\)=Qϕ​\(x<t,a\)−μtσt\+ϵ,μt=1\|𝒜t\|​∑a∈𝒜tQϕ​\(x<t,a\),\\bar\{Q\}\_\{t\}\(a\)=\\frac\{Q\_\{\\phi\}\(x\_\{<t\},a\)\-\\mu\_\{t\}\}\{\\sigma\_\{t\}\+\\epsilon\},\\qquad\\mu\_\{t\}=\\frac\{1\}\{\|\\mathcal\{A\}\_\{t\}\|\}\\sum\_\{a\\in\\mathcal\{A\}\_\{t\}\}Q\_\{\\phi\}\(x\_\{<t\},a\),\(7\)whereσt\\sigma\_\{t\}is the standard deviation over valid actions and𝒜t=𝒜​\(x<t\)\\mathcal\{A\}\_\{t\}=\\mathcal\{A\}\(x\_\{<t\}\)\. We then form a bounded value bonus,

bt​\(a\)=clip​\(β​Q¯t​\(a\),−c,c\),a∈𝒜t,b\_\{t\}\(a\)=\\mathrm\{clip\}\\left\(\\beta\\bar\{Q\}\_\{t\}\(a\),\-c,c\\right\),\\qquad a\\in\\mathcal\{A\}\_\{t\},\(8\)and add it to the base logits,

ℓ~t​\(a\)=ℓt​\(a\)\+bt​\(a\),a∈𝒜t,\\tilde\{\\ell\}\_\{t\}\(a\)=\\ell\_\{t\}\(a\)\+b\_\{t\}\(a\),\\qquad a\\in\\mathcal\{A\}\_\{t\},\(9\)with invalid actions masked before sampling\. The scalarβ\\betacontrols guidance strength, andccbounds the final logit bonus so that rare value outliers cannot dominate the base policy\. In the main experiments,β=0\.5\\beta=0\.5andc=5\.0c=5\.0\.

The completed molecule is then scored by the PMO oracle and handed back to the underlying optimizer exactly as in the baseline\. For PPO, the PPO loss and advantage objective are unchanged\. For REINVENT, AHC, and hill\-climbing, the likelihood update or sample\-selection rule is unchanged\. Q\-Steer therefore changes the behavior policy used during rollout, but it does not introduce a new optimizer objective or additional online oracle calls\.

Algorithm 1Q\-Steer rollout wrapper

Input:base logitsℓt\\ell\_\{t\}, frozen PAVS\-QQϕQ\_\{\\phi\}, guidance strengthβ\\beta, PMO oracle budgetBB\.

### 3\.5Connection to Doob\-Style Value Transforms

The logit tilt in Equation[9](https://arxiv.org/html/2607.26391#S3.E9)can be viewed as a practical finite\-vocabulary value transform\. If one had an exact positive desirability functionh​\(x<t\)h\(x\_\{<t\}\)for eventual success, a Doobhh\-transform would bias transitions by a ratio of future desirabilities, increasing the probability of trajectories that reach high\-value events\. In token generation, an action\-resolved analogue would favor actions whose continuations have higher conditional value than other actions at the same prefix\.

Q\-Steer should not be read as a theorem\-level Doob transform\. PAVS\-Q is learned, task\-specific, and used with normalization and clipping rather than as an exact harmonic function or exact success probability\. The connection is nevertheless useful because it explains the design: local transition probabilities are tilted by a frozen estimate of downstream value, while the expensive terminal oracle is still queried only after a full molecule is produced\.

### 3\.6Why Action\-Specific Values Matter

The method should only work if action\-specific values carry information beyond prefix quality\. A prefix\-broadcast valueV​\(x<t\)V\(x\_\{<t\}\)cannot change the relative next\-token probabilities at a fixed prefix, except through numerical side effects or interactions with clipping and masks\. In contrast,Q​\(x<t,a\)Q\(x\_\{<t\},a\)can increase the probability of one valid action while decreasing another from the same prefix\. A shuffled\-action control preserves the marginal distribution of value magnitudes but breaks the mapping between actions and predicted downstream rewards\. These controls test whether gains come from meaningful action\-value structure rather than generic logit scaling, reward\-model calibration, or reduced entropy alone\.

## 4Experiments

### 4\.1PMO23 Tasks and Online Budget

We evaluate on PMO23 with a 10,000\-call online oracle budget per run\(Gaoet al\.,[2022](https://arxiv.org/html/2607.26391#bib.bib6)\)\. PMO23 includes similarity, MPO, rediscovery, scaffold, isomer, SMARTS, QED, and QSAR tasks\. Each method is run with three seeds unless otherwise noted\. Appendix[A](https://arxiv.org/html/2607.26391#A1)gives the full task mapping, software stack, policy priors, optimizer settings, PAVS\-Q training defaults, and Q\-Steer rollout configuration\.

### 4\.2Baselines, Backbones, and Guided Settings

The primary study is a controlled factorial comparison across four LSTM optimizer families: PPO, REINVENT, AHC, and hill\-climbing\. Each family is evaluated in its baseline form and with Q\-Steer rollout steering, using the same PMO23 tasks, seeds, policy backbone, and 10,000\-call online oracle budget\. We denote the main guided setting asqsteer\_b05, corresponding toβ=0\.5\\beta=0\.5\. We additionally run the same full factorial design with a GPT2 backbone as a robustness replication\. The LSTM study remains the clean controlled setting for the main causal comparison, while GPT2 tests whether the average\-reward pattern survives a transformer molecular policy\.

### 4\.3Metrics

We report mean score, top\-10 mean score, cumulative top\-10 AUC, best score, validity, uniqueness, and runtime\. Unless otherwise stated, score summaries are computed over valid unique molecules produced within the 10,000\-call online budget; validity is the fraction of generated molecules marked valid, and uniqueness is the fraction of valid molecules that are unique\. Mean score therefore measures average valid\-unique quality rather than a duplicate\-weighted or invalid\-as\-zero reward\. Top\-10 and cumulative top\-10 AUC measure upper\-tail discovery and early discovery dynamics\.

### 4\.4Implementation and Run Accounting

All primary runs use ACEGEN\-aligned tokenization so that policy action IDs and PAVS\-Q action values refer to the same tokens\. PAVS\-Q is trained offline for each PMO task and then frozen during online optimization\. During rollout, values are computed on candidate actions in chunks, centered and scaled over valid actions, clipped to a bounded logit bonus, and added to logits withβ=0\.5\\beta=0\.5for the main setting\. The optimizer\-specific update rule, reward computation, and online oracle budget are unchanged\.

All online comparisons count only the molecules scored during the optimizer run and use the same 10,000\-call PMO oracle budget per task and seed\. PAVS\-Q is trained before the online optimization run and is frozen during evaluation; it does not receive gradients, rewards, or generated molecules from the online trajectory being evaluated\. Thus, Q\-Steer changes the sampling distribution but not the online oracle budget\. We report PAVS\-Q training as an offline prior, analogous to using a pretrained policy prior, and separate this offline cost from the online PMO budget\. The policy tokenizer and PAVS\-Q tokenizer are aligned by construction, and action\-value scores are applied only to the current policy action IDs\.

All aggregate comparisons use the paired seed intersection between a baseline optimizer and its Q\-Steer counterpart\. Tables report the number of paired seeds used for each task, and macro summaries report how many tasks have all three paired seeds\. The completed LSTM factorial design contains4×2×23×3=5524\\times 2\\times 23\\times 3=552successful formal runs, and the matched GPT2 replication contains another 552 successful formal runs\. A run is included only when the runner exits successfully and a completescores\.csvexists\. Infrastructure failures such as MolScore server port collisions or GPU out\-of\-memory errors are logged and retried rather than counted as method outcomes\. Appendix[A\.8](https://arxiv.org/html/2607.26391#A1.SS8)gives the run ledger, fixed\-online\-budget accounting, and paired\-seed aggregation rules; Appendix[B](https://arxiv.org/html/2607.26391#A2)records the paired seed count for each task, optimizer family, and backbone\.

## 5Results

### 5\.1Main Result: 8/8 Backbone–Optimizer Cells Improve Mean Reward

Table[1](https://arxiv.org/html/2607.26391#S5.T1)gives the PMO23 factorial summary\. The table is organized around the central causal comparison: within each backbone and optimizer family, the baseline and Q\-Steer runs use the same tasks, seeds, and 10,000\-call online oracle budget, and differ only in whether rollout logits receive action\-value guidance\. Across 1104 successful formal PMO23 runs, all eight backbone–optimizer cells have positive mean valid\-unique reward deltas\. In the controlled LSTM study, Q\-Steer improves mean score on 78/92 optimizer–task comparisons: 19/23 tasks for PPO and hill\-climbing, and 20/23 tasks for REINVENT and AHC\. The matched GPT2 replication adds another 77/92 task wins, with positive macro gains for every optimizer family\.

Table 1:PMO23 paired\-seed macro summary over valid unique molecules\. Each row compares a baseline optimizer to the same optimizer with Q\-Steer at the same backbone, task set, seed set, and 10,000\-call online oracle budget\. Deltas are Q\-Steer minus baseline\. Mean, top\-10, and AUC wins count tasks with positive paired\-seed deltas; higher is better except for uniqueness, where negative values indicate reduced diversity\.The LSTM rows are the cleanest controlled evidence: macro mean\-score gains range from \+0\.0343 to \+0\.0490, while all four optimizer families use the same policy backbone, PMO23 tasks, seeds, and online oracle budget\. This matters because the effect is not confined to PPO\. The same steering layer improves mean reward when attached to REINVENT, AHC, and hill\-climbing, so Q\-Steer is better understood as a sampling\-time primitive that composes with multiple molecular optimizers rather than as an optimizer\-specific objective modification\.

The main table also separates the strength and the boundary of the result\. Mean reward is the consistent gain; top\-10 wins and best\-score effects are mixed, and uniqueness decreases\. The mechanism test in Table[2](https://arxiv.org/html/2607.26391#S5.T2)is therefore central: it tests whether the mean\-reward improvement comes from action\-specific future\-value guidance rather than from generic logit perturbation or entropy reduction\.

### 5\.2Backbone and Optimizer Robustness

The matched GPT2 replication tests whether the pattern is tied to an LSTM molecular policy\. It preserves the same direction for every optimizer family: GPT2 PPO, REINVENT, AHC, and hill\-climbing all have positive macro mean\-score gains, ranging from \+0\.0330 to \+0\.0440, with 18–20 mean\-score task wins\. GPT2 also shows clearer cumulative top\-10 AUC gains than LSTM PPO, with AUC wins on 16–17 of 23 tasks across the four optimizer families\. We interpret this as backbone robustness for the average\-reward claim, not as evidence that every transformer architecture or scale will behave identically\.

Task\-bootstrap confidence intervals and exact sign tests are reported in Appendix Table[16](https://arxiv.org/html/2607.26391#A2.T16), and the complete PMO23 task\-level paired deltas for all eight backbone–optimizer comparisons are in Appendix[B](https://arxiv.org/html/2607.26391#A2)\. All eight task\-bootstrap intervals forΔ\\DeltaMean remain positive, and all exact sign tests favor Q\-Steer atp≤0\.001p\\leq 0\.001\. This supports the average\-reward claim at the PMO23 task level rather than only at the run aggregate level\.

The upper\-tail metrics are more nuanced\. Top\-10 wins are mixed for all families, and best\-score wins are weaker still\. Cumulative top\-10 AUC improves clearly for the GPT2 rows and for the LSTM non\-PPO optimizers, but is approximately neutral for LSTM PPO\. Appendix Table[11](https://arxiv.org/html/2607.26391#A1.T11)reports the secondary macro metrics, including top\-10, best\-score, validity, and uniqueness deltas\. These results prevent an overbroad claim: Q\-Steer is best described as average\-reward steering, not as a universally dominant best\-molecule discovery algorithm\.

### 5\.3Reward\-Diversity Tradeoff

The consistent reward gain comes with a consistent diversity cost\. Uniqueness decreases for every optimizer family and backbone\. In LSTM, the macro uniqueness deltas are \-0\.2101 for PPO, \-0\.0176 for REINVENT, \-0\.0164 for AHC, and \-0\.0927 for hill\-climbing; in GPT2, the corresponding deltas are smaller but still negative\. This supports the mechanism\-level interpretation that Q\-Steer concentrates probability mass around actions with high predicted future value\. It should therefore be treated as an exploitation knob, not a diversity\-preserving exploration mechanism\.

Figure[2](https://arxiv.org/html/2607.26391#S5.F2)summarizes the LSTM macro mean\-score gains across optimizer families together with the LSTM task\-level reward\-diversity tradeoff\. The full LSTM and GPT2 task\-level tables are reported in Appendix[B](https://arxiv.org/html/2607.26391#A2)\.

![Refer to caption](https://arxiv.org/html/2607.26391v1/x1.png)Figure 2:LSTM full\-factorial summary\. Panel \(a\) shows that all four LSTM optimizer families gain macro mean reward from Q\-Steer at the same 10,000\-call online budget\. Panel \(b\) shows PMO23 task\-level paired deltas; reward gains often coincide with lower uniqueness, consistent with exploitation\-oriented steering rather than diversity\-preserving exploration\. The matched GPT2 replication is summarized in Table[1](https://arxiv.org/html/2607.26391#S5.T1)\.
### 5\.4Mechanism: Action\-Specific Values

The decisive ablation is not another optimizer variant; it is whetherQ​\(x<t,a\)Q\(x\_\{<t\},a\)must be action\-specific\. Table[2](https://arxiv.org/html/2607.26391#S5.T2)compares true Q\-Steer to prefix\-broadcast and shuffled\-action controls in the LSTM PPO setting\. True action\-specific PAVS\-Q improves mean score on 19/23 tasks with macroΔ\\DeltaMean=\+0\.0343=\+0\.0343\. Prefix\-broadcast values, which preserve prefix quality but remove action discrimination, are nearly neutral: 12/23 wins andΔ\\DeltaMean=\+0\.0022=\+0\.0022\. Shuffled\-action values, which preserve value magnitudes but assign them to the wrong tokens, hurt performance: 3/23 wins andΔ\\DeltaMean=−0\.0459=\-0\.0459\.

Table 2:LSTM PPO mechanism controls on PMO23\. Deltas are versus LSTM PPO under paired seeds\. Prefix\-broadcast removes action discrimination; shuffled\-action breaks the mapping between values and action identities\. Shuffled\-action has four tasks with fewer than three paired seeds; the appendix records the partial count\.This ablation is central to the story because the ordering is qualitative, not just quantitative: correct action values help, state\-level broadcast values are almost inert, and wrong action assignments actively harm\. The useful signal is therefore not just a score scale, logit noise, or regularizer; it is the action\-specific mapping from candidate token to predicted downstream value\.

We further test this interpretation with a focused beta/shuffling diagnostic on a six\-task PMO subset using LSTM PPO, three seeds, and 1,500 online oracle calls per run\. Table[3](https://arxiv.org/html/2607.26391#S5.T3)gives dose\-response evidence, and Appendix Table[15](https://arxiv.org/html/2607.26391#A2.T15)reports the task\-level diagnostic deltas\. True action\-value steering wins 5/6 tasks at each tested strength, and the macro mean gain increases from \+0\.0586 atβ=0\.25\\beta=0\.25to about \+0\.10 atβ=0\.5\\beta=0\.5and1\.01\.0\. The cost also grows: uniqueness drops from \-0\.1201 to roughly \-0\.21 as guidance strengthens\. In contrast, shuffled action values lose 0/6 tasks and reduce mean score by \-0\.0624\. This supports the mechanism in Section 3: Q\-Steer works by using the correct action\-value assignment to concentrate rollout probability mass, not by adding arbitrary logit noise or merely reducing entropy\.

Table 3:Focused LSTM PPO mechanism diagnostic on six PMO tasks with three seeds and 1,500 online oracle calls per run\. Deltas are versus PPO base\. True action values improve mean score across steering strengths, while shuffled action values harm performance despite preserving value magnitudes\.
### 5\.5Runtime and Practical Cost

Because PAVS\-Q is an offline prior, Q\-Steer is not compute\-free: it adds model\-side action scoring during rollout\. Wall\-clock comparisons are implementation\-dependent: PPO steering is slower in our runs, while some non\-PPO guided runs are faster because they were executed under different accelerated infrastructure than the original CPU\-heavy baselines\. Appendix Tables[12](https://arxiv.org/html/2607.26391#A1.T12)and[13](https://arxiv.org/html/2607.26391#A1.T13)report runtime and cluster profiles for transparency rather than as an efficiency claim\. The scientific comparison is fixed\-online\-oracle performance; the factorial results show that extra model\-side computation can improve average valid\-unique reward under that budget, while runtime optimization remains necessary for deployment\.

## 6Discussion

#### Scope and next steps\.

The main result is that a frozen prefix\-action value model improves average valid\-unique reward across four optimizer families and two molecular language\-model backbones under a fixed online oracle budget\. This supports Q\-Steer as a reusable rollout steering primitive rather than a PPO\-specific modification\. The GPT2 replication reduces the risk of an LSTM\-only artifact, while broader transformer scaling remains future work\. The cost is additional offline/model\-side computation, lower uniqueness, and mixed upper\-tail gains; a natural next step is to combine action\-value guidance with exploration\-preserving mechanisms such as novelty\-aware penalties or adaptiveβ\\betaschedules\.

## 7Conclusion

Q\-Steer turns delayed molecular rewards into local action guidance\. By training PAVS\-Q to estimate prefix\-action future values and adding bounded normalized value bonuses to rollout logits, Q\-Steer improves average valid\-unique score without changing the optimizer update or online oracle budget\. Complete PMO23 factorial studies with LSTM and GPT2 backbones show robust mean\-score gains across PPO, REINVENT, AHC, and hill\-climbing\. This supports action\-value steering as a reusable primitive for oracle\-limited molecular policy optimization and motivates future work on combining it with exploration\-preserving mechanisms\.

## Broader Impacts

This work targets oracle\-limited molecular generation\. It may improve sample efficiency when oracle evaluations are expensive, but the same steering mechanism could be paired with objectives that favor unsafe or undesirable molecular properties\. Practical deployment should therefore use domain\-specific safety filters, constrained objectives, and expert review rather than treating benchmark reward maximization as sufficient for real\-world molecule design\.

## Limitations

This study has several limitations\. We evaluate LSTM and GPT2 backbones, but not broad transformer scaling or architecture\-universal dominance\. Uniqueness decreases across all optimizer families and both backbones, so Q\-Steer should be viewed as exploitation\-oriented unless paired with explicit diversity mechanisms\. The method adds model\-side computation because PAVS\-Q must be evaluated during rollout, and PAVS\-Q is a task\-specific offline prior whose cost is separate from the matched online PMO budget\. Top\-10 and best\-score improvements are mixed, so Q\-Steer is not a universal top\-k discovery enhancer\. Understanding how to amortize or transfer the value model across objectives remains future work\. Finally, the mechanism controls are run in LSTM PPO; extending them to every optimizer and backbone would further strengthen the mechanistic interpretation\.

## References

- T\. Blaschke, J\. Arús\-Pous, H\. Chen, C\. Margreitter, C\. Tyrchan, O\. Engkvist, K\. Papadopoulos, and A\. Patronov \(2020\)REINVENT 2\.0: an AI tool for de novo drug design\.Journal of Chemical Information and Modeling60\(12\),pp\. 5918–5922\.External Links:[Document](https://dx.doi.org/10.1021/acs.jcim.0c00915),[Link](https://doi.org/10.1021/acs.jcim.0c00915)Cited by:[§1](https://arxiv.org/html/2607.26391#S1.p1.1),[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px1.p1.1)\.
- GuacaMol: benchmarking models for de novo molecular design\.Journal of Chemical Information and Modeling59\(3\),pp\. 1096–1108\.External Links:[Document](https://dx.doi.org/10.1021/acs.jcim.8b00839),[Link](https://doi.org/10.1021/acs.jcim.8b00839)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Dathathri, A\. Madotto, J\. Lan, J\. Hung, E\. Frank, P\. Molino, J\. Yosinski, and R\. Liu \(2020\)Plug and play language models: a simple approach to controlled text generation\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=H1edEyBKDS)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Gao, T\. Fu, J\. Sun, and C\. W\. Coley \(2022\)Sample efficiency matters: a benchmark for practical molecular optimization\.InAdvances in Neural Information Processing Systems 35,pp\. 21342–21357\.Note:NeurIPS 2022 Datasets and Benchmarks TrackExternal Links:2206\.12411,[Document](https://dx.doi.org/10.48550/arXiv.2206.12411),[Link](https://arxiv.org/abs/2206.12411)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2607.26391#S4.SS1.p1.1)\.
- S\. K\. Gottipati, B\. Sattarov, S\. Niu, Y\. Pathak, H\. Wei, S\. Liu, S\. Liu, S\. Blackburn, K\. Thomas, C\. Coley, J\. Tang, S\. Chandar, and Y\. Bengio \(2020\)Learning to navigate the synthetically accessible chemical space using reinforcement learning\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 3668–3679\.External Links:[Link](https://proceedings.mlr.press/v119/gottipati20a.html)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px1.p1.1)\.
- N\. S\. Keskar, B\. McCann, L\. R\. Varshney, C\. Xiong, and R\. Socher \(2019\)CTRL: a conditional transformer language model for controllable generation\.arXiv preprint arXiv:1909\.05858\.External Links:1909\.05858,[Document](https://dx.doi.org/10.48550/arXiv.1909.05858),[Link](https://arxiv.org/abs/1909.05858)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Krause, A\. D\. Gotmare, B\. McCann, N\. S\. Keskar, S\. Joty, R\. Socher, and N\. F\. Rajani \(2021\)GeDi: generative discriminator guided sequence generation\.InFindings of the Association for Computational Linguistics: EMNLP 2021,Punta Cana, Dominican Republic,pp\. 4929–4952\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.424),[Link](https://aclanthology.org/2021.findings-emnlp.424/)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Liu, M\. Sap, X\. Lu, S\. Swayamdipta, C\. Bhagavatula, N\. A\. Smith, and Y\. Choi \(2021\)DExperts: decoding\-time controlled text generation with experts and anti\-experts\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),Online,pp\. 6691–6706\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.522),[Link](https://aclanthology.org/2021.acl-long.522/)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Olivecrona, T\. Blaschke, O\. Engkvist, and H\. Chen \(2017\)Molecular de\-novo design through deep reinforcement learning\.Journal of Cheminformatics9\(1\),pp\. 48\.External Links:[Document](https://dx.doi.org/10.1186/s13321-017-0235-x),[Link](https://doi.org/10.1186/s13321-017-0235-x)Cited by:[§1](https://arxiv.org/html/2607.26391#S1.p1.1),[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.External Links:1707\.06347,[Document](https://dx.doi.org/10.48550/arXiv.1707.06347),[Link](https://arxiv.org/abs/1707.06347)Cited by:[§1](https://arxiv.org/html/2607.26391#S1.p1.1),[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Yang and D\. Klein \(2021\)FUDGE: controlled text generation with future discriminators\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Online,pp\. 3511–3535\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.276),[Link](https://aclanthology.org/2021.naacl-main.276/)Cited by:[§2](https://arxiv.org/html/2607.26391#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AExperimental Specification

This appendix provides the implementation and accounting details behind the main PMO23 factorial results\. The intent is to make clear which quantities are part of the online molecular optimization budget, which quantities are offline priors, and how the paired comparisons in the main text were constructed\.

### A\.1Software Stack and Benchmark Interface

All online molecular optimization runs use ACEGEN\-open as the molecular RL framework\. The local ACEGEN package identifies itself as version 1\.1 and depends on PyTorch/TorchRL/TensorDict, RDKit, MolScore, and PromptSMILES\. The experiments use MolScore task presets to instantiate PMO and GuacaMol objectives\. Each online run is launched through an ACEGEN optimizer script and writes a run record plus the ACEGENscores\.csvfile used for aggregation\.

Table 4:Software components used for the online PMO experiments\.
### A\.2PMO23 Task Set

The PMO23 suite spans similarity, MPO, rediscovery, scaffold, isomer, SMARTS, QED, and QSAR\-style objectives\. Most tasks are loaded through MolScoreMolOpt:\*presets; the three rediscovery tasks useGuacaMol:\*presets\. All formal factorial runs use a 10,000\-call online oracle budget per task, seed, backbone, optimizer, and setting\.

Table 5:PMO23 task names used in the factorial study\.albuterol\_similarityamlodipine\_mpocelecoxib\_rediscoverydeco\_hopdrd2fexofenadine\_mpogsk3bisomers\_c9h10n2o2pf2clc7h8n2o2jnk3median\_molecules\_1median\_molecules\_2mestranol\_similarityosimertinib\_mpoperindopril\_mpoqedranolazine\_mposcaffold\_hopsitagliptin\_mpothiothixene\_rediscoverytroglitazone\_rediscoveryvalsartan\_smartszaleplon\_mpo

### A\.3Policy Backbones and Priors

The main controlled study uses ACEGEN’s defaultlstmmodel, which loads the ChEMBL vocabulary andlstm\_chembl\.ckptprior\. The LSTM actor has an embedding size of 256, hidden size of 512, three recurrent layers, and no dropout in the default ACEGEN factory\. The robustness replication uses ACEGEN’s defaultgpt2model with the Enamine REAL vocabulary andgpt2\_enamine\_real\.ckptprior\. The GPT2 factory uses a HuggingFace GPT2\-style configuration with 24 layers, 16 attention heads, embedding size 128, context length 2048, and dropout 0\.1\. These priors are not retrained before the PMO runs; the optimizer updates them online according to its own algorithm\.

Table 6:Policy backbone priors used in the factorial study\.
### A\.4Optimizer Hyperparameters

Table[7](https://arxiv.org/html/2607.26391#A1.T7)lists the optimizer\-level hyperparameters inherited from the ACEGEN denovo configurations, plus the overrides used for the PMO10k factorial runs\. PPO runs disable experience replay and use one PPO epoch in the formal matrix for throughput and stability; the PPO loss, clipping objective, and advantage computation remain otherwise unchanged\. For GPT2 PPO, generalized advantage estimation uses unshifted transformer sequence critics; recurrent LSTM policies keep shifted GAE under the same automatic configuration\.

Table 7:Optimizer configuration used in the PMO23 factorial study\. Batch parallelism was adjusted for hardware scheduling, but the online oracle budget was fixed at 10,000 scored molecules per run\.
### A\.5PAVS\-Q Training and Frozen Use

Each PMO task uses a task\-specific offline PAVS\-Q checkpoint\. Training starts from scored SMILES data, splits molecules into train and validation sets with a 0\.2 validation fraction, and decomposes every scored molecule into prefix\-action examples\. The supervised target is the standardized terminal task score\. The default PAVS\-Q architecture uses token embeddings of dimension 128, a one\-layer GRU prefix encoder with hidden size 128, an action embedding projection, and an MLP head with layer normalization, GELU nonlinearities, and dropout 0\.10\. Training uses AdamW with learning rate2×10−32\\times 10^\{\-3\}, weight decay10−410^\{\-4\}, batch size 512, eight epochs, gradient clipping at 5\.0, and checkpoint selection by held\-out validation metric\. During online PMO evaluation, the selected checkpoint is frozen and does not receive online rewards, gradients, or generated molecules from the evaluated trajectory\.

Table 8:PAVS\-Q value\-model defaults\. These are offline prior\-building settings, not online PMO oracle\-budget settings\.
### A\.6Q\-Steer Rollout Configuration

In the formal guided setting, the PAVS\-Q hook and rollout sampling mode are enabled in the ACEGEN configuration\. Table[9](https://arxiv.org/html/2607.26391#A1.T9)lists the rollout settings used for the main guided condition\. The candidate action set is the policy’s ACEGEN action vocabulary after the current valid\-action mask is applied\. Q\-Steer changes the sampled action distribution during rollout but does not add auxiliary losses, KL anchors, reward shaping, or optimizer\-specific objective changes in the formal factorial runs\.

Table 9:Q\-Steer rollout configuration for the formal guided PMO23 factorial setting\.The mechanism controls use the same machinery with different rollout controls\. The prefix\-broadcast control replaces all action\-specific values under a prefix by the valid\-action mean value, preserving prefix quality but removing action discrimination\. The shuffled\-action control rolls action values across the action dimension, preserving the marginal value magnitudes while breaking the action identity\. The beta diagnostic usesrollout\_betavalues 0\.25, 0\.5, and 1\.0 on a six\-task subset with 1,500 online oracle calls per run\.

### A\.7Metrics and Aggregation Details

Aggregates are computed from the ACEGENscores\.csvfile for each successful run\. The score column is selected from available task\-score columns such asscore,single,gmean, orvalid\_score\. Unless otherwise stated, mean score, top\-10 score, best score, and cumulative top\-10 AUC are computed over valid unique molecules\. Validity is the fraction of generated rows marked valid\. Uniqueness is the fraction of valid molecules marked unique\. Cumulative top\-10 AUC is computed by walking through available scoring steps, taking the mean of the best up to 10 valid unique scores observed so far, and averaging this curve over steps\.

All comparisons are paired by task and seed\. A task\-level delta is the mean over paired seeds for that task\. Macro deltas are then averages over PMO23 tasks, not over individual molecules\. A formal run is included only if its runner record has return code 0 and a completescores\.csvexists\. Failed infrastructure runs are retried and are not counted as method outcomes\.

### A\.8Run Ledger and Online\-Budget Accounting

Table[10](https://arxiv.org/html/2607.26391#A1.T10)summarizes the formal run accounting used by the paper\. The main factorial evidence contains 1104 successful 10,000\-call PMO23 runs: 552 for the controlled LSTM study and 552 for the matched GPT2 replication\. The focused beta/shuffling diagnostic is smaller by design and is used only for mechanism evidence\. These counts are reported to separate completed experimental evidence from infrastructure retries and exploratory pilots\.

Table 10:Formal run ledger\. A successful run requires runner return code 0 and a complete ACEGENscores\.csv\. The online oracle budget is the number of scored molecules available to the optimizer during evaluation; offline PAVS\-Q training is not counted as online PMO budget\.The paired comparisons keep the online PMO budget fixed\. Q\-Steer does not query the PMO oracle when scoring candidate next tokens; it evaluates only the frozen PAVS\-Q guide\. The completed molecule is still scored once by the same PMO oracle and then returned to the same optimizer update\. Thus the scientific comparison is fixed\-online\-oracle performance, not equal wall\-clock or equal offline\-pretraining cost\. This mirrors the usual use of pretrained molecular priors: offline prior construction is reported separately from the online benchmark budget\.

### A\.9Secondary Metrics and Validity

Table[11](https://arxiv.org/html/2607.26391#A1.T11)reports secondary macro metrics for the same paired PMO23 comparisons as Table[1](https://arxiv.org/html/2607.26391#S5.T1)\. These metrics are included to make the tradeoffs explicit\. The mean\-score gains are not accompanied by a uniform upper\-tail improvement: top\-10 and best\-score effects are mixed, especially in the LSTM rows\. Validity is approximately unchanged for PPO but decreases in several non\-PPO settings, while uniqueness decreases in every backbone–optimizer cell\. This supports the narrower interpretation used in the main text: Q\-Steer is an exploitation\-oriented average\-reward steering method, not a diversity\-preserving or universally best\-molecule\-improving method\.

Table 11:Secondary PMO23 macro metrics under the same paired comparisons as Table[1](https://arxiv.org/html/2607.26391#S5.T1)\. Deltas are Q\-Steer minus baseline, averaged over task\-level paired\-seed means\. Wins count tasks with positive deltas\.
### A\.10Wall\-Clock Cost Accounting

Q\-Steer adds model\-side computation because candidate actions are scored by PAVS\-Q during rollout\. Table[12](https://arxiv.org/html/2607.26391#A1.T12)reports wall\-clock accounting for the LSTM formal runs where runner seconds were available\. These numbers should not be interpreted as a hardware\-normalized efficiency benchmark: some baseline and guided non\-PPO runs were executed under different scheduling and acceleration conditions\. The robust conclusion is instead qualitative and conservative: Q\-Steer preserves the online oracle budget but can increase model\-side compute, and deployment would require engineering optimization of batched action scoring and caching\.

Table 12:LSTM formal\-run wall\-clock accounting\. Ratios are Q\-Steer divided by baseline runner seconds from paired runs\. Non\-PPO ratios are reported for transparency but not used as scientific efficiency claims because hardware and scheduling differed across some runs\.
### A\.11Compute Resources

The formal experiments were run on an internal Slurm cluster using the resource profiles in Table[13](https://arxiv.org/html/2607.26391#A1.T13)\. Runner records store per\-run wall\-clock seconds, and the runtime accounting above summarizes the successful LSTM formal runs\. The full research project used additional exploratory pilots and infrastructure retries; these were useful for engineering but are not counted as method outcomes unless the final runner exited successfully and produced a completescores\.csv\. GPU jobs requested one accelerator from the cluster’sgeneral\-gpupartition; exact accelerator model varied by node and scheduling, so runtime is reported as practical accounting rather than a hardware\-normalized efficiency benchmark\.

Table 13:Compute resource profiles used for the reported experiments\. Time is the Slurm wall\-time limit per array task; each individual PMO run also records elapsed seconds in its runner record\.
### A\.12Mechanism Diagnostic Scope

The main mechanism controls in Table[2](https://arxiv.org/html/2607.26391#S5.T2)use the LSTM PPO setting over PMO23\. The focused beta/shuffling diagnostic in Table[3](https://arxiv.org/html/2607.26391#S5.T3)uses six tasks, three seeds, five modes, and 1,500 online oracle calls per run\. This diagnostic is not intended to replace the full 10,000\-call PMO23 factorial evidence; it is included to isolate whether action\-specific value assignments, rather than generic logit perturbations, are responsible for the observed steering behavior\.

## Appendix BFull Task\-Level Tables

This appendix reports the task\-level paired comparisons underlying the macro summaries in the main text, along with task\-bootstrap confidence intervals and exact sign tests for the main macro effects\. Each row uses the paired seed intersection available for that task and comparison\.

Table 14:Full PMO23 macro summary across LSTM and GPT2 backbones\. Deltas are averaged over task\-level paired\-seed means\.Table 15:Focused LSTM PPO beta/shuffling mechanism diagnostic on six PMO tasks with three seeds and 1,500 online oracle calls per run\. Entries are task\-levelΔ\\DeltaMean versus PPO base\. The scaffold task is saturated at mean score 1\.0 for all modes\.Table 16:Task\-bootstrap uncertainty for PMO23 mean\-score effects\. Confidence intervals resample tasks with replacement; exact sign\-testppvalues test whether task\-level mean\-score deltas are symmetrically positive and negative after removing ties\.Table 17:Full PMO23 paired comparison: LSTM PPO baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 18:Full PMO23 paired comparison: LSTM REINVENT baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 19:Full PMO23 paired comparison: LSTM AHC baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 20:Full PMO23 paired comparison: LSTM Hill\-climbing baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 21:Full PMO23 paired comparison: GPT2 PPO baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 22:Full PMO23 paired comparison: GPT2 REINVENT baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 23:Full PMO23 paired comparison: GPT2 AHC baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 24:Full PMO23 paired comparison: GPT2 Hill\-climbing baseline versus Q\-Steer\. Deltas are Q\-Steer minus the same optimizer baseline\.Table 25:PMO23 mechanism controls\. Deltas are relative to LSTM PPO base under paired seeds\.