Policy Gradient Steering: Interventions from Behavioral Objectives

# Policy Gradient Steering: Interventions from Behavioral Objectives
Source: [https://arxiv.org/html/2607.27574](https://arxiv.org/html/2607.27574)
###### Abstract

Activation steering has emerged in large language models as a lightweight alternative for dynamically changing a model’s behavior at inference time\. However, we show that existing steering methods fail to steer even a simple policy in a two\-route gridworld environment\. To address this limitation, we propose Policy Gradient Steering \(PGS\), which formulates steering as a reinforcement learning problem\. PGS accumulates gradients of a temporary behavioral objective over a small set of rollouts or demonstrations to construct a removable task vector\. We first demonstrate the calibration and reversibility of PGS in a two\-route gridworld environment\. Using chess puzzles, we then evaluate independently fitted PGS vectors both in isolation and in combination, finding that compatible tactical objectives accumulate constructively\. Finally, in competitive football, we show that PGS can alter specific team behaviors and that its effects transfer across opponents\. Together, these results show that policy gradients provide a natural interface for constructing temporary and composable behavioral adaptations across diverse decision\-making domains\.

## 1Introduction

Trained policies are typically optimized for objectives fixed during training\. Although some policies expose controls conditioned on preferences or returns\(Abelset al\.[2019](https://arxiv.org/html/2607.27574#bib.bib249); Chenet al\.[2021](https://arxiv.org/html/2607.27574#bib.bib259)\), a fixed deployed policy may provide no such interface\. After deployment, however, we may want to impose a new behavioral preference, such as favoring one valid strategy over another, without retraining or permanently modifying the base policy\. The challenge is to turn a small set of trajectories scored under this temporary objective into a controllable intervention that can be applied, scaled, and removed at inference time\.

Existing activation\-steering methods commonly construct interventions from representations associated with contrasting examples or rollout outcomes\(Rimskyet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib256); Miaoet al\.[2026](https://arxiv.org/html/2607.27574#bib.bib238)\)\. This construction can fail for feed\-forward policies such as MLPs, whose pre\-decision activations depend only on the current observation\. When trajectories diverge from the same observation, their activations differ only after they reach different states\. As our two\-route gridworld shows, the resulting contrast may encode downstream state visitation without controlling the branching action\.

To address this limitation, we introduce Policy Gradient Steering \(PGS\), which uses policy gradients to translate behavioral feedback on trajectories into an additive activation intervention\. Rather than contrasting representations reached after outcomes diverge, PGS assigns credit to the actions that produced those outcomes and aggregates the resulting activation gradients into a single steering vector\. The vector can be scaled, composed, and removed at inference time while the base policy remains frozen\.

![Refer to caption](https://arxiv.org/html/2607.27574v1/x1.png)Figure 1:Overview of the policy gradient steering method\. Activation gradients are accumulated from a small batch of scored trajectories to construct an additive steering vector\.PGS occupies a middle ground between activation steering and policy fine\-tuning\. Like contrastive activation methods, it provides lightweight inference\-time control without modifying the base policy\(Rimskyet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib256); Miaoet al\.[2026](https://arxiv.org/html/2607.27574#bib.bib238)\)\. Like policy fine\-tuning and learned adaptations such as ReFT and LoRA, however, it derives the intervention by optimizing the behavioral objective\(Xieet al\.[2021](https://arxiv.org/html/2607.27574#bib.bib265); Wuet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib241); Huet al\.[2022](https://arxiv.org/html/2607.27574#bib.bib270)\)\. This raises a further question: whether independently constructed behavioral interventions can be combined without retraining, as explored for parameter and activation interventions in prior work\(Ilharcoet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib214); Pfeifferet al\.[2021](https://arxiv.org/html/2607.27574#bib.bib266); Stolfoet al\.[2025](https://arxiv.org/html/2607.27574#bib.bib267)\)\. We study this compositional property directly in chess\.

We evaluate PGS across three settings that expose distinct challenges\. A two\-route gridworld isolates the action\-credit failure of outcome\-based contrasts\. Chess tests whether independently fitted tactical interventions retain their effects under composition and across policies of different skill levels\. Competitive football extends the evaluation to multi\-agent policies, where an intervention’s effect may depend on the controlled policy and its strategic context\.

#### Contributions\.

1. 1\.Policy Gradient Steering, a method that converts arbitrary scalar feedback on trajectories into a removable activation intervention by assigning credit to the policy’s actions rather than to post\-outcome representations\.
2. 2\.A controlled gridworld analysis showing that outcome\-based activation contrasts can encode states reached after a decision without controlling the decision itself, supported by comparisons with contrastive steering methods and learned adaptations\.
3. 3\.An empirical evaluation across chess and competitive football examining how compact behavioral interventions compose across objectives and how their effects depend on the policy context\.

## 2Background

### 2\.1Post\-Training Behavioral Adaptation

For a trained policyπθ0​\(a∣o\)\\pi\_\{\\theta\_\{0\}\}\(a\\mid o\), we first consider behavioral adaptations that rely on additional optimization\.

#### Fine\-tuning and parameter task vectors\.

In reinforcement learning, policy fine\-tuning has been studied as online learning with access to a reference policy\(Xieet al\.[2021](https://arxiv.org/html/2607.27574#bib.bib265)\)\. Our fine\-tuning baseline updates policy weights by continuing optimization from the pretrained parameters under the newly specified behavioral objective\. When the base and adapted policies share an architecture, task arithmetic represents the effect of this optimization as a parameter difference\(Ilharcoet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib214)\):

τb=θb−θ0,\\tau\_\{b\}=\\theta\_\{b\}\-\\theta\_\{0\},\(1\)whereθb\\theta\_\{b\}denotes the fine\-tuned weights\. The resulting task vector can be applied with strengthα\\alphaas

θ​\(α\)=θ0\+α​τb\.\\theta\(\\alpha\)=\\theta\_\{0\}\+\\alpha\\tau\_\{b\}\.\(2\)This representation permits the adaptation to be scaled, negated, or combined with other parameter\-space adaptations\.

#### Low\-rank tuning\.

Low\-rank tuning restricts adaptation to a compact set of task\-specific parameters\. LoRA learns low\-rank updates to the policy weights\(Huet al\.[2022](https://arxiv.org/html/2607.27574#bib.bib270)\), whereas ReFT freezes the weights and learns a low\-rank intervention on hidden representations\(Wuet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib241)\)\. In our LoRA baseline, a task\-specific adapter is fitted and its effective weight\-space update is treated as a scalable, composable task\-specific adaptation, following the task\-arithmetic treatment of parameter deltas\(Ilharcoet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib214)\)\.

#### Preference adaptation\.

One way to support changing preferences at runtime is to train a policy that conditions its behavior on an explicit preference variable\(Abelset al\.[2019](https://arxiv.org/html/2607.27574#bib.bib249)\)\. Successor features similarly separate expected reward features from task\-specific weights, allowing adaptation to a new linear reward when the feature basis is known\(Dayan[1993](https://arxiv.org/html/2607.27574#bib.bib246); Barretoet al\.[2018](https://arxiv.org/html/2607.27574#bib.bib247); Borsaet al\.[2019](https://arxiv.org/html/2607.27574#bib.bib248)\)\. When a fixed policy exposes no such interface, an alternative is to construct an adaptation after training\. In language models, DPO adapts a policy from preferred–rejected pairs\(Rafailovet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib242)\), and Preference Vectors derive a scalable parameter\-space adaptation direction from such feedback\(Lianget al\.[2026](https://arxiv.org/html/2607.27574#bib.bib244)\)\.

### 2\.2Activation Steering Baselines

These methods construct activation interventions from positive and negative examples without using behavioral\-return gradients through the policy\.

#### Contrastive activation addition \(CAA\)\.

Given positive and negative example sets𝒟\+\\mathcal\{D\}\_\{\+\}and𝒟−\\mathcal\{D\}\_\{\-\}, CAA adds their mean activation difference at layerll\(Rimskyet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib256)\):

hl←hl\+α​\(𝔼𝒟\+​\[hl\]−𝔼𝒟−​\[hl\]\)\.h^\{l\}\\leftarrow h^\{l\}\+\\alpha\\left\(\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\+\}\}\[h^\{l\}\]\-\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\-\}\}\[h^\{l\}\]\\right\)\.\(3\)wherehlh^\{l\}denotes the activation at layerll, andα\\alphacontrols the intervention strength\.

#### K\-Steer\.

K\-Steer trains a nonlinear multi\-label classifier to recognize behavioral attributes from hidden activations\(Oozeeret al\.[2025](https://arxiv.org/html/2607.27574#bib.bib237)\)\. Given a classifiercϕc\_\{\\phi\}and a lossℒ\\mathcal\{L\}that rewards desired attributes and penalizes attributes to avoid, it updates the current activation as

hl←hl−α​∇hlℒ​\(cϕ​\(hl\)\)\.h^\{l\}\\leftarrow h^\{l\}\-\\alpha\\nabla\_\{h^\{l\}\}\\mathcal\{L\}\\\!\\left\(c\_\{\\phi\}\(h^\{l\}\)\\right\)\.\(4\)

#### COAST\.

COAST, introduced for vision\-language\-action models, fits conceptorsCsuccessC\_\{\\mathrm\{success\}\}andCfailureC\_\{\\mathrm\{failure\}\}from activations collected during successful and failed rollouts\(Miaoet al\.[2026](https://arxiv.org/html/2607.27574#bib.bib238)\)\. It applies the resulting contrastive conceptor as

hl←\[\(1−α\)​I\+α​\(Csuccess∧\(I−Cfailure\)\)\]​hl,h^\{l\}\\leftarrow\\left\[\(1\-\\alpha\)I\+\\alpha\\left\(C\_\{\\mathrm\{success\}\}\\wedge\(I\-C\_\{\\mathrm\{failure\}\}\)\\right\)\\right\]h^\{l\},\(5\)where∧\\wedgedenotes conceptor intersection\.

## 3Policy Gradient Steering

### 3\.1Problem Formulation

Letπ​\(a∣o\)\\pi\(a\\mid o\)be a pretrained policy that maps an observationooto a distribution over actions\. After training, we introduce a temporary behavioral objective through a scalar rewardrbr\_\{b\}\.

For a trajectoryτ=\(o0,a0,…,oT,aT\)\\tau=\(o\_\{0\},a\_\{0\},\\ldots,o\_\{T\},a\_\{T\}\), the behavioral return associated with the decision at timettis

Gt=∑k=tTγk−t​rb​\(ok,ak\),G\_\{t\}=\\sum\_\{k=t\}^\{T\}\\gamma^\{k\-t\}r\_\{b\}\(o\_\{k\},a\_\{k\}\),\(6\)whereγ∈\[0,1\]\\gamma\\in\[0,1\]controls how later outcomes are assigned to earlier decisions\.

Given a small dataset of reward\-scored trajectories𝒟=\{τ1,…,τN\}\\mathcal\{D\}=\\\{\\tau\_\{1\},\\ldots,\\tau\_\{N\}\\\}collected by a behavior policyμ\\mu, the steering problem is to obtain a controlled policyπ~\\widetilde\{\\pi\}that maximizes

maxπ~⁡𝔼τ∼π~​\[G​\(τ\)\]\.\\max\_\{\\widetilde\{\\pi\}\}\\;\\mathbb\{E\}\_\{\\tau\\sim\\widetilde\{\\pi\}\}\\\!\\left\[G\(\\tau\)\\right\]\.\(7\)We writeG​\(τ\)=G0G\(\\tau\)=G\_\{0\}for the trajectory return\. Classical contrastive steering formulation is recovered by splitting𝒟\\mathcal\{D\}equally into positive and negative trajectories and assigning each trajectory the scalar rewardrb​\(τ\)=\+1r\_\{b\}\(\\tau\)=\+1orrb​\(τ\)=−1r\_\{b\}\(\\tau\)=\-1, respectively\.

### 3\.2Constructing the PGS Vector

At a chosen activationhh, PGS applies the likelihood\-ratio construction underlying REINFORCE and the policy\-gradient theorem\(Williams[1992](https://arxiv.org/html/2607.27574#bib.bib231); Suttonet al\.[1999](https://arxiv.org/html/2607.27574#bib.bib233)\): it accumulates gradients of the observed actions weighted by their behavioral returns,

vPGS≔𝔼𝒟​\[ρt​\(Gt−b​\(ot\)\)​∇hlog⁡π​\(at∣ot\)\],v\_\{\\mathrm\{PGS\}\}\\coloneqq\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\left\[\\rho\_\{t\}\\bigl\(G\_\{t\}\-b\(o\_\{t\}\)\\bigr\)\\nabla\_\{h\}\\log\\pi\(a\_\{t\}\\mid o\_\{t\}\)\\right\],\(8\)Unlike policy\-gradient fine\-tuning, Equation[8](https://arxiv.org/html/2607.27574#S3.E8)differentiates with respect to the selected activation and averages these decision\-level gradients into a single state\-independent offset, which can be stored and applied without modifying the policy weights\. Once its steering strengthα\\alphahas been calibrated, inference applies the resulting vector as

h←h\+α​vPGS\.h\\leftarrow h\+\\alpha v\_\{\\mathrm\{PGS\}\}\.\(9\)For data collected by behavior policyμ\\mu, the per\-decision importance ratio is\(Degriset al\.[2012](https://arxiv.org/html/2607.27574#bib.bib232)\)

ρt=π​\(at∣ot\)μ​\(at∣ot\)\.\\rho\_\{t\}=\\frac\{\\pi\(a\_\{t\}\\mid o\_\{t\}\)\}\{\\mu\(a\_\{t\}\\mid o\_\{t\}\)\}\.\(10\)It is treated as fixed when computing the gradient and equals one for on\-policy data\. For off\-policy data, this simple per\-decision ratio corrects the action distribution only at the observations sampled underμ\\mu; it does not in general correct the mismatch between the state distributions of the behavior and target policies\. We therefore treat it as an approximate off\-policy construction rather than a generally unbiased estimator of Equation[7](https://arxiv.org/html/2607.27574#S3.E7)\. We use an action\-independent baselineb​\(ot\)b\(o\_\{t\}\)to reduce return variance without changing the expected on\-policy gradient\(Williams[1992](https://arxiv.org/html/2607.27574#bib.bib231)\)\.

### 3\.3Steering Algorithm

Algorithm[1](https://arxiv.org/html/2607.27574#alg1)summarizes fitting\. Inference applies the fitted intervention in the policy’s forward pass using Equation[9](https://arxiv.org/html/2607.27574#S3.E9)\. To calibrate its strength, we use the activation\-space Fisher matrix

F=𝔼o∼𝒟a∼π\(⋅∣o\)​\[∇hlog⁡π​\(a∣o\)​∇hlog⁡π​\(a∣o\)⊤\]\.F=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}o\\sim\\mathcal\{D\}\\\\ a\\sim\\pi\(\\cdot\\mid o\)\\end\{subarray\}\}\\left\[\\nabla\_\{h\}\\log\\pi\(a\\mid o\)\\nabla\_\{h\}\\log\\pi\(a\\mid o\)^\{\\top\}\\right\]\.\(11\)Under a local quadratic approximation to the action KL \(Appendix[D](https://arxiv.org/html/2607.27574#A4)\), the coefficient for a policy\-change budgetε\\varepsilonis

α=2​εvPGS⊤​F​vPGS\.\\alpha=\\sqrt\{\\frac\{2\\varepsilon\}\{v\_\{\\mathrm\{PGS\}\}^\{\\top\}Fv\_\{\\mathrm\{PGS\}\}\}\}\.\(12\)We calibrate the intervention from the scalar directional curvaturevPGS⊤​F​vPGSv\_\{\\mathrm\{PGS\}\}^\{\\top\}Fv\_\{\\mathrm\{PGS\}\}\. We discuss trust\-region calibration and natural\-gradient preconditioning as future extensions in Section[7](https://arxiv.org/html/2607.27574#S7)\.

Algorithm 1Policy Gradient SteeringInput: policyπ\\pi, trajectories𝒟\\mathcal\{D\}, behavioral rewardrbr\_\{b\}, activation sitell, KL budgetε\\varepsilon Output: PGS intervention\(vPGS,α\)\(v\_\{\\mathrm\{PGS\}\},\\alpha\)

1:Compute returns

GtG\_\{t\}from

rbr\_\{b\}
2:Estimate the baseline

b​\(ot\)b\(o\_\{t\}\)
3:Accumulate

vPGSv\_\{\\mathrm\{PGS\}\}at site

llusing Equation[8](https://arxiv.org/html/2607.27574#S3.E8)

4:Estimate the directional curvature

vPGS⊤​F​vPGSv\_\{\\mathrm\{PGS\}\}^\{\\top\}Fv\_\{\\mathrm\{PGS\}\}
5:

α←2​ε/\(vPGS⊤​F​vPGS\)\\alpha\\leftarrow\\sqrt\{2\\varepsilon/\(v\_\{\\mathrm\{PGS\}\}^\{\\top\}Fv\_\{\\mathrm\{PGS\}\}\)\}
6:return

\(vPGS,α\)\(v\_\{\\mathrm\{PGS\}\},\\alpha\)

## 4Limits of Existing Activation Steering

### 4\.1Gridworld Protocol

Figure[2](https://arxiv.org/html/2607.27574#S4.F2)shows the two\-route environment\. The policy is trained only to navigate from the start state to the goal; route preference is introduced after training\.

For the trained MLP policy, layerllhas pre\-activation and post\-ReLU representation

hprel\\displaystyle h\_\{\\mathrm\{pre\}\}^\{l\}=Wl​hpostl−1\+bl,\\displaystyle=W^\{l\}h\_\{\\mathrm\{post\}\}^\{l\-1\}\+b^\{l\},\(13\)hpostl\\displaystyle h\_\{\\mathrm\{post\}\}^\{l\}=ReLU⁡\(hprel\),\\displaystyle=\\operatorname\{ReLU\}\(h\_\{\\mathrm\{pre\}\}^\{l\}\),\(14\)withhpost0=oh\_\{\\mathrm\{post\}\}^\{0\}=o\. An activation intervention inserts a functionglg\_\{l\}before the remaining policy layers while leavingWlW^\{l\}andblb^\{l\}fixed\. CAA, COAST, K\-Steer, and the random activation baseline intervene athpost2h\_\{\\mathrm\{post\}\}^\{2\}\. PGS jointly fits offsets athpost1h\_\{\\mathrm\{post\}\}^\{1\}andhpost2h\_\{\\mathrm\{post\}\}^\{2\}and normalizes their concatenation as one direction\. These sites are fixed before fitting, while intervention strength is selected using validation data only\. PGS uses one gradient update and fine\-tuning uses three updates on the same fixed batch\.

To adapt the contrastive baselines to route steering, we treat activations from successful target\-route trajectories as positive and those from successful alternative\-route trajectories as negative, excluding failures\. Every valid timestep inherits its trajectory label, and we do not balance the sets by subsampling\. CAA contrasts the two set means, COAST fits a conceptor to each set, and K\-Steer uses a class\-balanced classifier loss\.

![Refer to caption](https://arxiv.org/html/2607.27574v1/x2.png)\(a\)Navigation training
![Refer to caption](https://arxiv.org/html/2607.27574v1/x3.png)\(b\)Route preference

Figure 2:Two\-route Gridworld used to isolate behavioral preference from task completion\. In \(a\), the policy learns to navigate from the start square to the goal\. In \(b\), the upper route is preferred and the lower route is discouraged\.
### 4\.2Gridworld Results

#### Activation steering\.

We first fit each method using trajectories sampled from the frozen policy\. Existing activation steering baselines remain close to the unsteered route distribution\. PGS instead shifts the route preference in the requested direction while preserving path efficiency\.

Table 1:Gridworld steering fitted from frozen\-policy rollouts\.
#### Parameterization\.

We next compare intervention parameterizations without selecting application strength from behavioral returns\. For each fitted direction, we estimate its local Fisher curvature on the fit states and choose the scale for a shared target action\-KL budget\. PGS is the strongest compact realization, while fine\-tuning is strongest overall\. ReFT and LoRA produce smaller route shifts, and all parameterizations retain comparable path lengths\. PGS, ReFT, and fine\-tuning achieve the target KL closely\. LoRA is less stable because one fitted direction has near\-zero local curvature\. Across the tested budgets, PGS’s measured held\-out KL closely tracks the requested budget \(separate appendix\)\.

Table 2:Gridworld parameterization comparison at a common target KL budget of0\.10\.1\. Application scales are computed from each fitted direction’s local Fisher curvature without a behavioral scale sweep\. Cells report mean±\\pmsample standard deviation over fit seeds\.

### 4\.3Why Contrastive Steering Fails

We isolate the failure using only two demonstrations: one successful trajectory through each route\. Even in this extreme low\-data regime, PGS shifts the route preference in both directions, whereas the contrastive baselines remain close to the unsteered policy\. Under this two\-demonstration fit, PGS produces the strongest top\-route shift while fine\-tuning is stronger in the reverse direction\.

Table 3:Gridworld steering fitted from one demonstration per route\.The failure follows from the information available at the route decision\. At the branching observation, both demonstrations induce the same hidden representationhh\. The decision\-local contrast is therefore

vCAAbranch=h\+−h−=h−h=0\.v\_\{\\mathrm\{CAA\}\}^\{\\mathrm\{branch\}\}=h^\{\+\}\-h^\{\-\}=h\-h=0\.\(15\)The fitted CAA baseline pools activations across all valid timesteps, so it may recover a nonzero contrast from route\-specific observations reached after the decision\. That contrast describes the consequences of the route choice rather than the action that causes it\. PGS can distinguish the two actions because∇hlog⁡π​\(a\+∣o\)\\nabla\_\{h\}\\log\\pi\(a^\{\+\}\\mid o\)and∇hlog⁡π​\(a−∣o\)\\nabla\_\{h\}\\log\\pi\(a^\{\-\}\\mid o\)differ even when their pre\-action representations are identical\.

## 5Evaluating Task Composition

### 5\.1Experimental Setup and Metrics

We use a frozen Maia\-1500 policy, one of the Maia models trained to predict human moves at a specified player rating\(McIlroy\-Younget al\.[2020](https://arxiv.org/html/2607.27574#bib.bib201)\)\. We draw puzzles from the open Lichess puzzle database\(Lichess\.org[2026](https://arxiv.org/html/2607.27574#bib.bib202)\)and select three tactical motifs: forks, pins, and skewers; Figure[3](https://arxiv.org/html/2607.27574#S5.F3)shows illustrative fork and pin puzzles\. For each puzzle, the policy receives a binary reward of one when it selects the canonical move and zero otherwise\. For each motif, we construct an independent adaptation from 40 training puzzles\. Fit, calibration, and test puzzles are disjoint\. Each adaptation is calibrated independently to a fixed action\-KL budget\. Compositions combine these isolated adaptations without an additional composition\-level rescaling, and we report the resulting composed KL\.

#### Canonical\-move likelihood\.

We evaluate each adaptation on held\-out puzzles using the negative log\-likelihood of the canonical move:

ℒm​\(S\)=−𝔼\(o,a⋆\)∼𝒟m​\[log⁡πS​\(a⋆∣o\)\],\\mathcal\{L\}\_\{m\}\(S\)=\-\\mathbb\{E\}\_\{\(o,a^\{\\star\}\)\\sim\\mathcal\{D\}\_\{m\}\}\\left\[\\log\\pi\_\{S\}\(a^\{\\star\}\\mid o\)\\right\],\(16\)whereSSdenotes the set of active adaptations\. Lower values indicate that the policy assigns greater probability to the canonical move\. For motifmm, we define the gain from a compositionSSasgm​\(S\)=ℒm​\(∅\)−ℒm​\(S\)g\_\{m\}\(S\)=\\mathcal\{L\}\_\{m\}\(\\varnothing\)\-\\mathcal\{L\}\_\{m\}\(S\)\.*Isolated gain*averagesgm​\(\{m\}\)g\_\{m\}\(\\\{m\\\}\)across motifs, while*composed gain*averagesgm​\(S\)g\_\{m\}\(S\)with all three adaptations active\.

#### Composition retention\.

To test whether composition sacrifices any individual objective, we report

Rmin=min\|S\|≥2m∈S⁡gm​\(S\)gm​\(\{m\}\)\.R\_\{\\min\}=\\min\_\{\\begin\{subarray\}\{c\}\|S\|\\geq 2\\\\ m\\in S\\end\{subarray\}\}\\frac\{g\_\{m\}\(S\)\}\{g\_\{m\}\(\\\{m\\\}\)\}\.\(17\)A value of one means that every objective preserves its isolated improvement; values above one indicate constructive interaction even for the least\-retained objective\.

#### Intervention cost\.

We additionally report the mean action\-distribution KL from the original policy to the three\-way composition and the number of scalars required to store the three adaptations\. Further implementation and calibration details are provided in the appendix\.

![Refer to caption](https://arxiv.org/html/2607.27574v1/x4.png)\(a\)Fork puzzle
![Refer to caption](https://arxiv.org/html/2607.27574v1/x5.png)\(b\)Pin puzzle

Figure 3:Examples of tactical motifs used to fit and compose independently calibrated behavioral adaptations\.

### 5\.2Results

#### Comparison with adaptation baselines\.

Table[4](https://arxiv.org/html/2607.27574#S5.T4)compares all methods using the same six residual blocks and behavioral\-change budget\. PGS achieves the strongest isolated gain and a composed gain comparable to LoRA and Fine\-tuning, while outperforming ReFT\. All methods preserve positive gains for every motif after composition, showing that the three tactical objectives are mutually compatible\. PGS obtains this performance with the smallest stored adaptation, supporting its use as a compact mechanism for constructing and composing behavioral interventions\.

Table 4:Matched\-support chess composition across all six residual blocks\. Values are mean±\\pmsample standard deviation across five fit seeds\.
#### Robustness across policy skill\.

We repeat the matched composition comparison with Maia\-1100 and Maia\-1900, using the same protocol as for Maia\-1500 and comparing every method against its corresponding frozen base policy\. This experiment tests whether the relative composition behavior is specific to one Maia checkpoint or persists across policies trained to imitate different player skill levels\. Every method improves all three checkpoints in isolation and after composition\. The leading baseline varies by checkpoint, while PGS remains close to the strongest composed result and retains its substantially smaller stored adaptation\. The composition effect is therefore not specific to Maia\-1500, but the experiment does not support a checkpoint\-independent ranking of adaptation methods\.

Table 5:Matched\-support chess composition across Maia checkpoints\. Values are mean±\\pmsample standard deviation across five fit seeds\.
#### Layer allocation\.

Table[6](https://arxiv.org/html/2607.27574#S5.T6)compares three ways of allocating the PGS adaptations across layers\. The same\-layer condition places every motif at the strongest common site according to its fit\-side gradient norm\. The gradient\-selected condition assigns each motif to a distinct layer to maximize the combined gradient\-norm criterion\. The permuted condition uses the same distinct layers but rotates their motif assignments\.

The comparison reveals a trade\-off between isolated strength and composition\. The common layer produces the strongest isolated adaptations, the gradient\-selected allocation gives the strongest minimum retention, and the permuted allocation gives the strongest composed gain with the lowest policy change\. Distributing adaptations across layers can therefore improve composition efficiency, but isolated gradient strength does not identify the best joint allocation\.

Table 6:PGS composition with all objectives at one layer or distributed across distinct layers\. Values are mean±\\pmsample standard deviation across five fit seeds under the exact expected canonical\-action objective\.

## 6Policy Diversification in Football

![Refer to caption](https://arxiv.org/html/2607.27574v1/figures/gfootball.png)Figure 4:Google Research Football environment used for the behavior\-steering study\(Kurachet al\.[2020](https://arxiv.org/html/2607.27574#bib.bib195)\)\.### 6\.1Experimental Setup

We study frozen team policies released bySonget al\.\([2024](https://arxiv.org/html/2607.27574#bib.bib277)\):defensive\_passer,flank\_pass,group\_pressure, andoffensive\_passer\. These policies implement distinct learned football strategies and serve as both controlled teams and opponents\. We ask whether PGS can produce temporary behavioral variants of these policies without retraining them\. Such variants could support more diverse populations for cross\-play evaluation, opponent curricula, or training leagues\(Bettiniet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib219)\)\.

We study three behavioral objectives: completed passes, possession regains, and shot attempts\. Each objective rewards its corresponding event, and we evaluate it using the matched held\-out event count and behavior rate\. The appendix protocol records the other configurable football objectives, which are outside the main study\.

Table 7:Unsteered full\-match main\-study behaviors; values are mean±\\pmsample standard deviation over five self\-play matches\.
### 6\.2Results

#### Passing Behavior\.

We use completed passing as the target behavior for the main comparison between PGS, ReFT, LoRA, Fine\-tuning, and a PGS\-norm random\-activation baseline\. All methods use full\-match self\-play trajectories and are calibrated to the same action\-KL budget\. Table[8](https://arxiv.org/html/2607.27574#S6.T8)reports full\-match outcomes for the unmodifiedgroup\_pressurepolicy and each intervention\. PGS produces a pronounced shift toward passing, yielding the highest mean pass\-attempt count and a completed\-pass count comparable to the strongest learned adaptation\. In contrast, random activation and Fine\-tuning remain near the unmodified policy, showing that neither intervention magnitude nor direct parameter optimization is sufficient to induce the target behavior at the same policy\-change budget\. Variation across held\-out matches nevertheless remains substantial\.

Table 8:Full\-match outcomes for shared interventions togroup\_pressurein self\-play\. Values are mean±\\pmsample standard deviation over held\-out match seeds\.PGS interventions also transfer across opponents\. We fit one intervention fromgroup\_pressurematches against each non\-self opponent, then evaluate each frozen intervention against all four policies\. Each intervention increases mean completed passes relative to the unmodified policy across evaluation opponents, while the intervention fitted from trajectories with greater passing exposure is consistently strongest\. Thus, PGS can encode a transferable team behavior, with its effectiveness shaped by the fitting trajectories\.

Table 9:Cross\-play completed passes from PGS applied togroup\_pressure\(mean±\\pmsample standard deviation over held\-out match seeds\)\.
#### Behavioral Diversity and Fitting Sensitivity\.

Beyond the main passing comparison, we apply PGS across multiple team policies and behavioral objectives\. We compare directions fitted from each target policy’s demonstrations \(Table[10](https://arxiv.org/html/2607.27574#S6.T10)\) with directions fitted from the policy strongest in the target behavior \(Table[11](https://arxiv.org/html/2607.27574#S6.T11)\)\. Neither fitting source dominates uniformly, showing that steering effectiveness depends on the interaction between the controlled policy, behavioral objective, and fitting trajectories\. This dependence is most pronounced for possession regains, a team\-level event requiring agent\-level credit assignment\. These results extend PGS beyond a single passing intervention while identifying fit\-data selection as a central challenge for reliable multi\-agent steering\.

Table 10:Absolute self\-play outcomes from PGS with target\-policy demonstrations from both sides \(mean±\\pmsample standard deviation over five held\-out match seeds\)\.Table 11:Absolute self\-play outcomes from PGS with offline source\-policy demonstrations from both sides \(mean±\\pmsample standard deviation over five held\-out match seeds\)\.

## 7Discussion

#### What PGS offers\.

PGS turns scalar behavioral feedback into a removable activation intervention by accumulating return\-weighted action\-score gradients from a fixed batch\. Its advantage is operational: it constructs a direction in one gradient estimation step, leaves the base\-policy parameters unchanged, and permits the same stored vector to be strengthened, weakened, or removed at inference time\. PGS is therefore suited to settings where low\-step, low\-storage, post\-training control is valuable, while iterative fine\-tuning, LoRA, and ReFT provide complementary routes for optimizing a selected behavioral objective\.

#### Reusable behavioral interventions\.

Chess and football expose complementary forms of reuse\. In chess, independently constructed PGS vectors retain their tactical effects under composition, although effective activation sites need not compose equally well\. In football, a fitted passing intervention transfers across opponents without refitting, although its effectiveness depends on the fitting trajectories\. Together, these results motivate libraries of reusable behavioral controls accompanied by composition and data\-selection rules\.

#### Evaluation boundaries\.

PGS is a local adaptation method: Fisher scaling is a local Fisher approximation, activation\-site choice remains consequential, and a behavioral effect does not by itself establish a semantic or mechanistic interpretation of the fitted vector\. Evaluations should therefore report behavioral gain together with achieved policy KL, task outcomes, seed\-to\-seed variation, storage, and inference cost\. Chess puzzle likelihood measures tactical preference rather than playing strength, while football target\-behavior gains must be read alongside their collateral effects and competitive\-match outcomes\.

#### Future work\.

PGS suggests three extensions\. Off\-policy fitting could use cumulative or marginalized importance weights, doubly robust policy\-gradient estimators\(Huang and Jiang[2020](https://arxiv.org/html/2607.27574#bib.bib235)\), or temporal\-difference gradient estimation\(Tosattoet al\.[2022](https://arxiv.org/html/2607.27574#bib.bib236)\)\. When the local Fisher approximation is inaccurate at finite scale, empirical action\-KL backtracking could refine the closed\-form coefficient, paralleling trust\-region optimization\(Schulmanet al\.[2015](https://arxiv.org/html/2607.27574#bib.bib234)\)while calibrating a fixed activation\-space direction\. Finally, natural\-gradient preconditioning withF−1​vPGSF^\{\-1\}v\_\{\\mathrm\{PGS\}\}\(Kakade[2001](https://arxiv.org/html/2607.27574#bib.bib250)\)could alter the direction rather than merely scale it\.

## 8Related Work

### 8\.1Post\-hoc Steering and Adaptation

Preference Goal Tuning \(PGT\) keeps a goal\-conditioned policy frozen and optimizes only its continuous goal embedding from trajectory preferences\(Zhaoet al\.[2026](https://arxiv.org/html/2607.27574#bib.bib243)\)\. Like PGS, it treats post\-training adaptation as inference\-time control rather than parameter fine\-tuning\. Unlike PGS, PGT requires a policy with a native goal\-conditioning interface and iteratively fits its control variable, whereas PGS constructs a removable activation vector from scalar behavioral returns for policies with no such interface\. Policy Gradient Guidance instead trains conditional and unconditional policy branches to expose a test\-time guidance scale\(Qiet al\.[2025](https://arxiv.org/html/2607.27574#bib.bib260)\); PGS constructs its intervention after the base policy has been trained\.

Activation\-steering methods derive interventions from contrastive examples, learned attributes, or rollout outcomes\(Zouet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib100); Liet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib261); Turneret al\.[2023](https://arxiv.org/html/2607.27574#bib.bib262); Rimskyet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib256); Stoehret al\.[2024](https://arxiv.org/html/2607.27574#bib.bib257); Oozeeret al\.[2025](https://arxiv.org/html/2607.27574#bib.bib237); Miaoet al\.[2026](https://arxiv.org/html/2607.27574#bib.bib238)\)\. PPLM uses gradients from an attribute model to perturb hidden states iteratively during decoding, while LatentSeek applies policy gradients to per\-instance latent sequences at test time\(Dathathriet al\.[2020](https://arxiv.org/html/2607.27574#bib.bib263); Liet al\.[2025](https://arxiv.org/html/2607.27574#bib.bib264)\); PGS instead aggregates return\-weighted action\-score gradients into a reusable, state\-independent vector\. ReFT and LoRA are related learned adaptation mechanisms\(Huet al\.[2022](https://arxiv.org/html/2607.27574#bib.bib270); Wuet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib241)\); we use them as baselines rather than claiming their parameterizations as contributions\.

### 8\.2Task Vectors and Composition

Task arithmetic composes fine\-tuning deltas, and early gradient steps can themselves form task vectors\(Ilharcoet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib214); Zhouet al\.[2026](https://arxiv.org/html/2607.27574#bib.bib240)\)\. AdapterFusion instead learns to combine separately trained task adapters\(Pfeifferet al\.[2021](https://arxiv.org/html/2607.27574#bib.bib266)\)\. Activation\-space composition has likewise been studied through dynamic multi\-property steering and compositional steering tokens\(Scalenaet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib268); Radevskiet al\.[2026](https://arxiv.org/html/2607.27574#bib.bib269)\)\. Preference Vectors use DPO to train models for preferred and label\-reversed objectives, then subtract their weights to form scalable task vectors\(Rafailovet al\.[2023](https://arxiv.org/html/2607.27574#bib.bib242); Lianget al\.[2026](https://arxiv.org/html/2607.27574#bib.bib244)\)\. Contrastive weight steering similarly constructs parameter\-space directions from opposing behaviors\(Fierro and Roger[2026](https://arxiv.org/html/2607.27574#bib.bib245)\)\. PGS does not claim novelty for the correspondence between gradients and task vectors\. Its focus is constructing removable activation\-space vectors from scalar action or trajectory returns and calibrating their individual and composed effects in policy space\.

### 8\.3Multi\-agent Adaptation

Explicit opponent models can learn opponent strategy patterns\(Heet al\.[2016](https://arxiv.org/html/2607.27574#bib.bib252)\); hierarchical opponent models can infer latent goals and support few\-shot adaptation to unseen policies\(Huanget al\.[2024](https://arxiv.org/html/2607.27574#bib.bib253)\)\. Our football study does not infer an explicit opponent model; it tests whether interventions fitted under one strategic context transfer or specialize across agent roles and opponents\. Shared\-policy MARL makes role specificity important: agents commonly act through shared policy parameters\(Yuet al\.[2022](https://arxiv.org/html/2607.27574#bib.bib271)\), yet representing distinct agent policies or tasks requires agent\-identifying information\(Terryet al\.[2020](https://arxiv.org/html/2607.27574#bib.bib272)\)\.

## 9Conclusion

We introduced Policy Gradient Steering, which turns temporary behavioral objectives into compact, removable activation interventions calibrated under a policy\-KL budget\. Across controlled route choice and compositional chess steering, our results show that policy\-gradient credit can support inference\-time behavioral adaptation without modifying the base policy; the football study extends this question to multi\-agent contexts\. PGS therefore provides a practical interface for introducing, combining, and removing behavioral preferences after training\.

## References

- A\. Abels, D\. M\. Roijers, T\. Lenaerts, A\. Nowé, and D\. Steckelmacher \(2019\)Dynamic weights in multi\-objective deep reinforcement learning\.InInternational Conference on Machine Learning,Vol\.97,pp\. 11–20\.External Links:[Link](https://proceedings.mlr.press/v97/abels19a.html)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px3.p1.1)\.
- J\. Ansel, E\. Yang, H\. He, N\. Gimelshein, A\. Jain, M\. Voznesensky, B\. Bao, P\. Bell, D\. Berard, E\. Burovski, G\. Chauhan, A\. Chourdia, W\. Constable, A\. Desmaison, Z\. DeVito, E\. Ellison, W\. Feng, J\. Gong, M\. Gschwind, B\. Hirsh, S\. Huang, K\. Kalambarkar, L\. Kirsch, M\. Lazos, M\. Lezcano, Y\. Liang, J\. Liang, Y\. Lu, C\. K\. Luk, B\. Maher, Y\. Pan, C\. Puhrsch, M\. Reso, M\. Saroufim, M\. Y\. Siraichi, H\. Suk, S\. Zhang, M\. Suo, P\. Tillet, X\. Zhao, E\. Wang, K\. Zhou, R\. Zou, X\. Wang, A\. Mathews, W\. Wen, G\. Chanan, P\. Wu, and S\. Chintala \(2024\)PyTorch 2: faster machine learning through dynamic python bytecode transformation and graph compilation\.Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2\.Cited by:[§E\.2](https://arxiv.org/html/2607.27574#A5.SS2.p1.1)\.
- Astral Software Inc \(2025\)uv\.Note:SoftwareExternal Links:[Link](https://github.com/astral-sh/uv)Cited by:[§E\.2](https://arxiv.org/html/2607.27574#A5.SS2.p1.1)\.
- A\. Barreto, D\. Borsa, J\. Quan, T\. Schaul, D\. Silver, M\. Hessel, D\. Mankowitz, A\. Zidek, and R\. Munos \(2018\)Transfer in deep reinforcement learning using successor features and generalised policy improvement\.InInternational Conference on Machine Learning,Vol\.80,pp\. 501–510\.External Links:[Link](https://proceedings.mlr.press/v80/barreto18a.html)Cited by:[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px3.p1.1)\.
- M\. Bettini, R\. Kortvelesy, and A\. Prorok \(2024\)Controlling behavioral diversity in multi\-agent reinforcement learning\.InInternational Conference on Machine Learning,Vol\.235,pp\. 3611–3636\.External Links:[Link](https://proceedings.mlr.press/v235/bettini24a.html)Cited by:[§6\.1](https://arxiv.org/html/2607.27574#S6.SS1.p1.1)\.
- D\. Borsa, A\. Barreto, J\. Quan, D\. J\. Mankowitz, H\. van Hasselt, R\. Munos, D\. Silver, and T\. Schaul \(2019\)Universal successor features approximators\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=S1VWjiRcKX)Cited by:[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px3.p1.1)\.
- L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. Mordatch \(2021\)Decision transformer: reinforcement learning via sequence modeling\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 15084–15097\.External Links:[Link](https://arxiv.org/abs/2106.01345)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p1.1)\.
- S\. Dathathri, A\. Madotto, J\. Lan, J\. Hung, E\. Frank, P\. Molino, J\. Yosinski, and R\. Liu \(2020\)Plug and play language models: a simple approach to controlled text generation\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=H1edEyBKDS)Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- P\. Dayan \(1993\)Improving generalization for temporal difference learning: the successor representation\.Neural Computation5\(4\),pp\. 613–624\.External Links:[Document](https://dx.doi.org/10.1162/neco.1993.5.4.613)Cited by:[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px3.p1.1)\.
- T\. Degris, M\. White, and R\. S\. Sutton \(2012\)Linear off\-policy actor\-critic\.InInternational Conference on Machine Learning,pp\. 179–186\.External Links:[Link](https://icml.cc/2012/papers/268.pdf)Cited by:[§3\.2](https://arxiv.org/html/2607.27574#S3.SS2.p1.3)\.
- C\. Fierro and F\. Roger \(2026\)Steering language models with weight arithmetic\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- C\. R\. Harris, K\. J\. Millman, S\. van der Walt, R\. Gommers, P\. Virtanen, D\. Cournapeau, E\. Wieser, J\. Taylor, S\. Berg, N\. J\. Smith, R\. Kern, M\. Picus, S\. Hoyer, M\. H\. van Kerkwijk, M\. Brett, A\. Haldane, J\. F\. del Río, M\. Wiebe, P\. Peterson, P\. Gérard\-Marchant, K\. Sheppard, T\. Reddy, W\. Weckesser, H\. Abbasi, C\. Gohlke, and T\. E\. Oliphant \(2020\)Array programming with numpy\.Nature585,pp\. 357–362\.External Links:[Document](https://dx.doi.org/10.1038/s41586-020-2649-2),[Link](https://doi.org/10.1038/s41586-020-2649-2)Cited by:[§E\.2](https://arxiv.org/html/2607.27574#A5.SS2.p1.1)\.
- H\. He, J\. Boyd\-Graber, K\. Kwok, and H\. Daumé III \(2016\)Opponent modeling in deep reinforcement learning\.InInternational Conference on Machine Learning,Vol\.48,pp\. 1804–1813\.External Links:[Link](https://proceedings.mlr.press/v48/he16.html)Cited by:[§8\.3](https://arxiv.org/html/2607.27574#S8.SS3.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px2.p1.1),[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- J\. Huang and N\. Jiang \(2020\)From importance sampling to doubly robust policy gradient\.InInternational Conference on Machine Learning,Vol\.119,pp\. 4434–4443\.External Links:[Link](https://proceedings.mlr.press/v119/huang20b.html)Cited by:[§7](https://arxiv.org/html/2607.27574#S7.SS0.SSS0.Px4.p1.1)\.
- Y\. Huang, A\. Liu, F\. Kong, Y\. Yang, S\. Zhu, and X\. Feng \(2024\)Efficient adaptation in mixed\-motive environments via hierarchical opponent modeling and planning\.InInternational Conference on Machine Learning,Vol\.235,pp\. 20004–20022\.External Links:[Link](https://proceedings.mlr.press/v235/huang24p.html)Cited by:[§8\.3](https://arxiv.org/html/2607.27574#S8.SS3.p1.1)\.
- J\. D\. Hunter \(2007\)Matplotlib: a 2d graphics environment\.Computing in Science & Engineering9\.Cited by:[§E\.2](https://arxiv.org/html/2607.27574#A5.SS2.p1.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2023\)Editing models with task arithmetic\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6t0Kwf8-jrj)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px1.p1.3),[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px2.p1.1),[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- S\. M\. Kakade \(2001\)A natural policy gradient\.InAdvances in Neural Information Processing Systems,Vol\.14\.Cited by:[§7](https://arxiv.org/html/2607.27574#S7.SS0.SSS0.Px4.p1.1)\.
- K\. Kurach, A\. Raichuk, P\. Stańczyk, M\. Zając, O\. Bachem, L\. Espeholt, C\. Riquelme, D\. Vincent, M\. Michalski, O\. Bousquet, and S\. Gelly \(2020\)Google research football: a novel reinforcement learning environment\.InProceedings of the Thirty\-Fourth AAAI Conference on Artificial Intelligence,pp\. 4501–4510\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i04.5878),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/5878)Cited by:[§E\.2](https://arxiv.org/html/2607.27574#A5.SS2.p1.1),[Figure 4](https://arxiv.org/html/2607.27574#S6.F4)\.
- H\. Li, C\. Li, T\. Wu, X\. Zhu, Y\. Wang, Z\. Yu, E\. H\. Jiang, S\. Zhu, Z\. Jia, Y\. N\. Wu, and Z\. Zheng \(2025\)Seek in the dark: reasoning via test\-time instance\-level policy gradient in latent space\.External Links:2505\.13308,[Link](https://arxiv.org/abs/2505.13308)Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2023\)Inference\-time intervention: eliciting truthful answers from a language model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 41451–41530\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/81b8390039b7302c909cb769f8b6cd93-Abstract-Conference.html)Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- R\. Liang, C\. Hsu, C\. Yu, S\. Agrawal, S\. Huang, C\. Lin, S\. Chen, K\. Huang, and S\. Sun \(2026\)Adaptive helpfulness–harmlessness alignment with preference vectors\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 1646–1668\.External Links:[Link](https://aclanthology.org/2026.eacl-long.77/)Cited by:[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px3.p1.1),[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- Lichess\.org \(2026\)Lichess Open Database: Puzzles\.Note:Accessed 2026\-07\-27External Links:[Link](https://database.lichess.org/#puzzles)Cited by:[§5\.1](https://arxiv.org/html/2607.27574#S5.SS1.p1.1)\.
- R\. McIlroy\-Young, S\. Sen, J\. Kleinberg, and A\. Anderson \(2020\)Aligning superhuman AI with human behavior: chess as a model system\.InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 1677–1687\.External Links:[Document](https://dx.doi.org/10.1145/3394486.3403219)Cited by:[§5\.1](https://arxiv.org/html/2607.27574#S5.SS1.p1.1)\.
- M\. M\. Miao, S\. Kim, B\. Yang, and L\. Ungar \(2026\)Contrastive conceptor activation steering \(coast\): unlocking vision\-language\-action models through hidden states\.External Links:2605\.17144Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p2.1),[§1](https://arxiv.org/html/2607.27574#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.27574#S2.SS2.SSS0.Px3.p1.2),[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- N\. F\. Oozeer, L\. Marks, F\. Barez, and A\. Abdullah \(2025\)Beyond linear steering: unified multi\-attribute control for language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 23513–23557\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1278),[Link](https://aclanthology.org/2025.findings-emnlp.1278/)Cited by:[§2\.2](https://arxiv.org/html/2607.27574#S2.SS2.SSS0.Px2.p1.2),[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- J\. Pfeiffer, A\. Kamath, A\. Rücklé, K\. Cho, and I\. Gurevych \(2021\)AdapterFusion: non\-destructive task composition for transfer learning\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,pp\. 487–503\.External Links:[Link](https://aclanthology.org/2021.eacl-main.39/)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p4.1),[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- Y\. Poupart \(2026\)LCZeroLens\.Note:Software, version 0\.4\.0External Links:[Link](https://github.com/Xmaster6y/lczerolens)Cited by:[§E\.2](https://arxiv.org/html/2607.27574#A5.SS2.p1.1)\.
- J\. Qi, H\. Tang, and Z\. Zhu \(2025\)Policy gradient guidance enables test time control\.InSecond Workshop on Aligning Reinforcement Learning Experimentalists and Theorists \(ARLET\),External Links:[Link](https://neurips.cc/virtual/2025/136153)Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p1.1)\.
- G\. Radevski, K\. Gashteovski, G\. Hong, C\. Lawrence, and G\. Glavaš \(2026\)Compositional steering of large language models with steering tokens\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,pp\. 31087–31104\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1435),[Link](https://aclanthology.org/2026.acl-long.1435/)Cited by:[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 53728–53747\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px3.p1.1),[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. Turner \(2024\)Steering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15504–15522\.External Links:[Link](https://aclanthology.org/2024.acl-long.828/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.828)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p2.1),[§1](https://arxiv.org/html/2607.27574#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.27574#S2.SS2.SSS0.Px1.p1.3),[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- D\. Scalena, G\. Sarti, and M\. Nissim \(2024\)Multi\-property steering of large language models with dynamic activation composition\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,pp\. 577–603\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.34),[Link](https://aclanthology.org/2024.blackboxnlp-1.34/)Cited by:[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- J\. Schulman, S\. Levine, P\. Abbeel, M\. Jordan, and P\. Moritz \(2015\)Trust region policy optimization\.InInternational Conference on Machine Learning,Vol\.37,pp\. 1889–1897\.External Links:[Link](https://proceedings.mlr.press/v37/schulman15.html)Cited by:[§7](https://arxiv.org/html/2607.27574#S7.SS0.SSS0.Px4.p1.1)\.
- Y\. Song, H\. Jiang, Z\. Tian, H\. Zhang, Y\. Zhang, J\. Zhu, Z\. Dai, W\. Zhang, and J\. Wang \(2024\)An empirical study on google research football multi\-agent scenarios\.Machine Intelligence Research21,pp\. 549–570\.External Links:[Document](https://dx.doi.org/10.1007/s11633-023-1426-8)Cited by:[§6\.1](https://arxiv.org/html/2607.27574#S6.SS1.p1.1)\.
- N\. Stoehr, K\. Du, V\. Snæbjarnarson, R\. West, R\. Cotterell, and A\. Schein \(2024\)Activation scaling for steering and interpreting language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 8189–8200\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.479/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.479)Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- A\. Stolfo, V\. Balachandran, S\. Yousefi, E\. Horvitz, and B\. Nushi \(2025\)Improving instruction\-following in language models through activation steering\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/8c3262a4c965ba9888f120d4f9e13478-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p4.1)\.
- R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour \(1999\)Policy gradient methods for reinforcement learning with function approximation\.InAdvances in Neural Information Processing Systems,Vol\.12\.External Links:[Link](https://proceedings.neurips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html)Cited by:[§3\.2](https://arxiv.org/html/2607.27574#S3.SS2.p1.1)\.
- J\. K\. Terry, N\. Grammel, S\. Son, B\. Black, and A\. Agrawal \(2020\)Revisiting parameter sharing in multi\-agent deep reinforcement learning\.External Links:2005\.13625,[Link](https://arxiv.org/abs/2005.13625)Cited by:[§8\.3](https://arxiv.org/html/2607.27574#S8.SS3.p1.1)\.
- The LCZero Authors \(2018\)LeelaChessZero\.Note:SoftwareExternal Links:[Link](https://github.com/LeelaChessZero/lc0)Cited by:[§B\.1](https://arxiv.org/html/2607.27574#A2.SS1.p1.1)\.
- S\. Tosatto, A\. Patterson, M\. White, and A\. R\. Mahmood \(2022\)A temporal\-difference approach to policy gradient estimation\.InInternational Conference on Machine Learning,Vol\.162,pp\. 21609–21632\.External Links:[Link](https://proceedings.mlr.press/v162/tosatto22a.html)Cited by:[§7](https://arxiv.org/html/2607.27574#S7.SS0.SSS0.Px4.p1.1)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2023\)Steering language models with activation engineering\.External Links:2308\.10248,[Link](https://arxiv.org/abs/2308.10248)Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8,pp\. 229–256\.Cited by:[§3\.2](https://arxiv.org/html/2607.27574#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2607.27574#S3.SS2.p1.5)\.
- Z\. Wu, A\. Arora, Z\. Wang, A\. Geiger, D\. Jurafsky, C\. D\. Manning, and C\. Potts \(2024\)ReFT: representation finetuning for language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/75008a0fba53bf13b0bb3b7bff986e0e-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px2.p1.1),[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.
- T\. Xie, N\. Jiang, H\. Wang, C\. Xiong, and Y\. Bai \(2021\)Policy finetuning: bridging sample\-efficient offline and online reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 27395–27407\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/e61eaa38aed621dd776d0e67cfeee366-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.27574#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.27574#S2.SS1.SSS0.Px1.p1.3)\.
- O\. Yadan \(2019\)Hydra \- a framework for elegantly configuring complex applications\.Note:GithubExternal Links:[Link](https://github.com/facebookresearch/hydra)Cited by:[§E\.2](https://arxiv.org/html/2607.27574#A5.SS2.p1.1)\.
- C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. Wu \(2022\)The surprising effectiveness of ppo in cooperative multi\-agent games\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§8\.3](https://arxiv.org/html/2607.27574#S8.SS3.p1.1)\.
- G\. Zhao, K\. Lian, H\. Ru, B\. Zhang, H\. Lin, Z\. Mu, H\. Fu, Q\. Fu, S\. Cai, Z\. Wang, and Y\. Liang \(2026\)Preference goal tuning: post\-training as latent control for frozen policies\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=TygL9qfa0d)Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p1.1)\.
- L\. Zhou, D\. Solombrino, D\. Crisostomi, M\. S\. Bucarelli, G\. A\. D’Inverno, F\. Silvestri, and E\. Rodolà \(2026\)On task vectors and gradients\.InProceedings of UniReps: the Third Edition of the Workshop on Unifying Representations in Neural Models,Proceedings of Machine Learning Research, Vol\.322,pp\. 398–417\.External Links:[Link](https://proceedings.mlr.press/v322/zhou26a.html)Cited by:[§8\.2](https://arxiv.org/html/2607.27574#S8.SS2.p1.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. Hendrycks \(2023\)Representation engineering: a top\-down approach to ai transparency\.External Links:2310\.01405Cited by:[§8\.1](https://arxiv.org/html/2607.27574#S8.SS1.p2.1)\.

Appendix

## Appendix AGridworld Additional Details

### A\.1Protocol

- •Freeze the policy before data collection and use disjoint fit, validation, and evaluation streams\.
- •Apply activation methods at declared post\-ReLU sites, rank\-one LoRA to all linear weights, and fit the optimized realizations with fixed\-batch updates\.
- •Fit PGS for one update and the optimized parameter\-space realizations for three updates, using learning rate10−210^\{\-2\}\. Fit PGS jointly ath1\_postandh2\_post, then normalize the concatenated activation\-offset vector to unit Euclidean norm\. Leave other optimized tensors unchanged and apply them at scaleα/10−2\\alpha/10^\{\-2\}\.
- •In the primary contrastive comparisons, treatα\\alphaas application strength and select it on validation data only\. In the parameterization comparison, instead computeα\\alphafrom directional Fisher curvature for a target action KL of0\.10\.1, without a behavioral scale sweep\.
- •Evaluate the selected object in both directions on the same held\-out stream and report unsuccessful routes explicitly\.

The successful\-route preference is

Rroute​\(τ\)=\{\+1,top\-route success,−1,bottom\-route success,rfail,failure\.R\_\{\\mathrm\{route\}\}\(\\tau\)=\\begin\{cases\}\+1,&\\text\{top\-route success\},\\\\ \-1,&\\text\{bottom\-route success\},\\\\ r\_\{\\mathrm\{fail\}\},&\\text\{failure\}\.\\end\{cases\}\(18\)

### A\.2Change Inspection

For both primary fit regimes, we reconstruct each seed\-specific validation\-selected PGS intervention for all five fit seeds, evaluate every action at every nonterminal state, and average probabilities across fitted seeds\. The plots expose changes hidden by route\-level aggregates without selecting a favorable fitted direction\.

![Refer to caption](https://arxiv.org/html/2607.27574v1/x6.png)Figure 5:Mean statewise action probabilities for the original policy and seed\-specific validation\-selected PGS fitted from 20 frozen\-policy rollouts\. Each row compares the original policy \(left\) with PGS \(right\)\. Blue bold text marks each panel’s greedy action; intervention probabilities are averaged over five fitted seeds\.![Refer to caption](https://arxiv.org/html/2607.27574v1/x7.png)Figure 6:Mean statewise action probabilities for the original policy and seed\-specific validation\-selected PGS fitted from one demonstration per route\. Each row compares the original policy \(left\) with PGS \(right\)\. Blue bold text marks each panel’s greedy action; intervention probabilities are averaged over five fitted seeds\.
### A\.3Ablations

All ablations reuse the rollout protocol and paired validation and evaluation streams\. Except for the learning\-rate sweep, each setting is selected on validation data before held\-out evaluation\.

#### Fit trajectory count\.

PGS produces a directional change even from the smallest fit batch, but its effect does not improve monotonically with additional trajectories\. Neither PGS nor fine\-tuning exhibits a uniform sample\-scaling trend over these small fit batches\.

Table 12:Gridworld fit trajectories ablation\. Cells report mean±\\pmsample standard deviation of disjoint test top\-route probability over five fit seeds\.
#### Return treatment\.

We compare raw returns, batch\-centered returns, standardized returns, and a fitted linear state baseline\. After normalizing the resulting steering direction, standardization is equivalent to centering, while the fitted baseline provides no clear behavioral improvement\. We therefore use the simple batch\-centered return in the main experiments\.

Table 13:Gridworld return treatment and fitted\-baseline comparison\.
#### Intervention site\.

PGS is sensitive to where the offset is applied\. Intervening at the first hidden layer is more effective than at the default second\-layer site, while jointly fitting offsets at both post\-ReLU sites produces the strongest route preference, with the two pre\-ReLU sites a close second\. We therefore use the joint post\-ReLU sites in the main comparison\.

Table 14:Gridworld intervention sites ablation using policy rollouts\. Alpha is selected separately for each setting on validation data; values report mean±\\pmsample standard deviation on the disjoint test stream over five fit seeds\.
#### Optimizer strength\.

After three fixed\-batch updates, unit\-normalized activation offsets are stable from learning rates10−410^\{\-4\}through10−210^\{\-2\}and weaken at larger rates\. Full\-policy deltas divided by their learning rate are stable through10−110^\{\-1\}but degrade sharply at learning rate11\. We fix the shared optimizer learning rate to10−210^\{\-2\}; application strength remains method\-specific and is selected separately\.

Table 15:Gridworld optimizer\-strength ablation\. Cells report the validation\-selectedα\\alphaand disjoint\-test top\-route probability over five fit seeds\.
#### Optimization steps\.

After unit normalization, the PGS direction is behaviorally stable from one to five fixed\-batch updates and changes little at ten\. Fine\-tuning is strongest after three updates and degrades with further reuse of the same batch\. We therefore use one update for PGS and three for fine\-tuning\.

Table 16:Gridworld updates ablation\. Cells report mean±\\pmsample standard deviation of disjoint test top\-route probability over five fit seeds\.

## Appendix BAdditional Chess Details

### B\.1Encoding and Legal Actions

LCZeroLens uses LCZero’s representation and represents each position as a current\-player\-oriented tensor with 112 planes of size8×88\\times 8\(The LCZero Authors[2018](https://arxiv.org/html/2607.27574#bib.bib198)\)\. The first 104 planes encode up to eight positions of history using twelve piece planes and one repetition plane per position; the remaining planes encode castling rights, side\-to\-move orientation, the half\-move clock, and constant metadata\. Positions with Black to move are flipped into the same canonical learner orientation\.

The policy head uses LCZero’s 1,858\-entry move encoding\. At every decision we mask logits for illegal moves and normalize the policy only over the legal set\. Dataset moves are stored in UCI notation and encoded relative to the current board orientation, with decoded promotions matched to their legal moves\. Fitting, validation, and evaluation use the same representation, orientation convention, and legal\-action mask\.

### B\.2Protocol

- •Partition puzzle identities globally into exclusive fork, pin, and skewer fit, validation, and evaluation cohorts\.
- •Optimize the exact expected binary reward of the canonical legal action at each fixed offline puzzle state\.
- •Compare channel PGS with rank\-four ReFT, rank\-four LoRA, and Fine\-tuning over all six residual blocks\. PGS and ReFT act after each block’s second convolution and ReLU, while LoRA and Fine\-tuning update the corresponding six convolutional weight tensors\.
- •Use one fitting update for PGS and three for ReFT, LoRA, and Fine\-tuning\.
- •Estimate directional curvature from the realized action KL at coefficients±10−3\\pm 10^\{\-3\}and set each independently fitted object’s coefficient toα=2​\(0\.1\)/\(v⊤​F​v\)\\alpha=\\sqrt\{2\(0\.1\)/\(v^\{\\top\}Fv\)\}\.
- •Add parameter deltas, apply ReFT residual maps in parallel, and sum PGS interventions when composing objectives\. The main composition does not apply an additional joint\-KL rescaling\.

### B\.3Ablations

All reported values are mean±\\pmsample standard deviation across five fit seeds\. Every condition uses the same disjoint calibration and test cohorts\.

#### Layer support\.

We compare nested bottom\-up and top\-down residual support, a policy\-head control, and all six residual blocks\. Each independently fitted object receives the same total isolated action\-KL budget, so adding sites does not increase the allowed policy change\. Bottom\-up support is consistently more effective than matched top\-down support\. PGS achieves its strongest composed gain with all six blocks, while the parameter\-update baselines saturate with four bottom\-up blocks\. Head\-only adaptation is weak for every method and produces negative composed gain for PGS, indicating that the promoted result depends on residual features rather than only the policy head\.

Table 17:Layer\-support ablation across the four matched methods\. Bottom\-up support starts at the earliest residual block, top\-down support starts at the latest block, and every independently fitted object receives one total isolated action\-KL budget of0\.10\.1\.
#### Fit\-data budget\.

The nested data\-budget study shows a clear low\-data regime with weaker and more variable composition\. Isolated gain, composed gain, and retention stabilize as the fit set grows, with no material composed\-gain improvement from doubling the promoted budget\. This supports 40 puzzles per motif as a reasonable fixed default without selecting it on test outcomes\.

Table 18:PGS fit\-data ablation using nested puzzle subsets \(each smaller budget is contained in the next\) with the promoted all\-six\-layer channel parameterization\. The 40\-puzzle setting is the frozen default; test outcomes do not select the budget\.
#### Optimization steps\.

Additional PGS updates trade isolated performance and policy\-change efficiency for stronger three\-way composition\. ReFT and LoRA improve more gradually, whereas Fine\-tuning is effectively unchanged across the tested update counts\. The promoted one\-update PGS configuration therefore represents the balanced isolated/composed setting rather than the maximum\-composition setting\. The baseline defaults remain frozen at three updates\.

Table 19:Update\-count sensitivity on matched all\-six\-layer support\. The promoted defaults are one update for PGS and three updates for ReFT, LoRA, and Fine\-tuning\.
#### PGS parameterization\.

Channel PGS is both substantially smaller and behaviorally stronger than the spatial control in isolated and composed gain\. Spatial PGS induces a smaller three\-way KL, but this accompanies much weaker target improvement rather than better composition at matched effectiveness\. We therefore retain channel PGS as the canonical parameterization\.

Table 20:PGS parameterization control over all six residual blocks\. Channel PGS stores one broadcast coefficient per channel and site; spatial PGS stores a coefficient at every channel and board location\.

## Appendix CAdditional Football Details

### C\.1Protocol

- •Collect complete unsteered matches against a source opponent and fit PGS from the controlled policy’s actions and returns\.
- •Keep fit, validation, and evaluation seeds disjoint; validation fixes the operating point under an action\-KL budget\.
- •Apply the same frozen artifact against its source and unseen opponents\.
- •Report paired held\-out goal difference, action KL, and event\-based behavior measures, with base and KL\-matched random controls\.

#### Behavior Objectives\.

The main football study fixes three outcomes: completed passes, possession regains, and shot attempts\. The implementation also supports possession, turnover avoidance, and wide attacking play as configurable objectives\. These alternatives are protocol options rather than main\-study outcomes; changing the objective requires a separately specified fit, validation, and held\-out evaluation\.

### C\.2Player\-Role Structure and Base Phenotype

#### Critic Player\-Role Bias\.

The frozen critic’s values vary systematically with the active player’s role inflank\_pass\(Figure[7](https://arxiv.org/html/2607.27574#A3.F7)\)\. Median values rise from center backs through fullbacks and central midfielders to wide midfielders and center forwards\. Raw critic magnitude therefore contains a strong role\-dependent offset and is not, by itself, evidence of opponent\-specific structure\. We remove this offset by centering critic values within each player role before constructing critic\-weighted directions\.

![Refer to caption](https://arxiv.org/html/2607.27574v1/x8.png)Figure 7:Frozen critic values forflank\_pass, grouped by the active player’s role\. The distributions reveal a pronounced positional bias in the uncentered critic output\.
#### Behavior Definitions and Base Phenotype\.

Table[21](https://arxiv.org/html/2607.27574#A3.T21)reports the four additional unsteered full\-match behaviors over five self\-play matches per policy: possession rate, turnovers, final\-third entries, and wide final\-third entries\. We compute these as possible behavioral targets\.

Table 21:Unsteered full\-match appendix behaviors; values are mean±\\pmsample standard deviation over five self\-play matches\.

### C\.3Opponent\-Conditioned Adaptation

This study asks whether prior interaction with an opponent can produce a useful temporary response from the frozengroup\_pressurepolicy\. PGS fits one temporary intervention fromgroup\_pressuretrajectories collected againstdefensive\_passer, then evaluates that intervention both againstdefensive\_passerand in self\-play\. Thus, opponent\-conditioned describes the interaction data that shape the intervention, rather than an explicit learned model of the opponent\.

Table 22:Opponent\-conditioned goal steering\. Values are mean±\\pmsample standard deviation over held\-out match seeds\.
### C\.4Ablations

#### Steering from Off\-Policy Trajectories\.

Table[23](https://arxiv.org/html/2607.27574#A3.T23)steers the frozengroup\_pressurepolicy using trajectories demonstrated bydefensive\_passeragainstgroup\_pressure\. It compares a ratio\-free off\-policy PGS surrogate with the same surrogate weighted by the per\-action likelihood ratio\. This policy\-pair\-specific diagnostic tests sensitivity to ratio weighting, rather than establishing an unbiased off\-policy estimator or a general football advantage\.

Table 23:Off\-policy completed\-passes steering\. Values are mean±\\pmsample standard deviation\.
#### Critic\-Based Credit Assignment\.

We compare five simple baselines for completed\-pass returns: no centering, centering over the fit trajectory, centering by player type within each collected batch, centering by player type after pooling the fit batches, and subtraction of an observation\-based ridge predictor fitted on the fit trajectories\. The last condition is an in\-sample learned baseline, not a cross\-fitted or causal critic\.

Table 24:Completed\-passes steering under simple reward\-weight baselines\. Values are mean±\\pmsample standard deviation over held\-out match seeds\.
#### Role\-Specific Steering\.

We compare a shared intervention with independently routed vectors by player role and player index\. Because these schemes allocate different numbers of vectors, this is a routing\-and\-capacity ablation rather than a pure test of role semantics\.

Table 25:Held\-out completed passes steering\. Values are mean±\\pmsample standard deviation over held\-out match seeds\.
#### Fit\-Data Budget\.

Table[26](https://arxiv.org/html/2607.27574#A3.T26)varies the self\-play data used to fit a shared completed\-passes intervention forgroup\_pressure\. The split\-data condition retains equal temporal halves from the left and right policy trajectories\. This descriptive ablation does not establish monotonic scaling or a confirmed behavioral effect\.

Table 26:Fit\-data\-budget ablation\. Values are mean±\\pmsample standard deviation\.
#### Discount Factor\.

Table[27](https://arxiv.org/html/2607.27574#A3.T27)varies the return discount factor while holding the self\-play fit budget at one match \(both policy sides; 6,000 steps\)\. This descriptive sweep does not select a discount factor or establish a confirmed behavioral effect\.

Table 27:Discount\-factor ablation\. Values are mean±\\pmsample standard deviation\.

## Appendix DActivation\-Site Calibration and Sensitivity

#### Euclidean sensitivity\.

LetJl​\(hl\)J\_\{l\}\(h\_\{l\}\)denote the PGS objective downstream of a candidate activation site and letql=∇hlJlq\_\{l\}=\\nabla\_\{h\_\{l\}\}J\_\{l\}\. For an additive interventionδl=α​ql\\delta\_\{l\}=\\alpha q\_\{l\}, a first\-order expansion predicts

Jl​\(hl\+δl\)−Jl​\(hl\)≈α​‖ql‖2\.J\_\{l\}\(h\_\{l\}\+\\delta\_\{l\}\)\-J\_\{l\}\(h\_\{l\}\)\\approx\\alpha\\left\\lVert q\_\{l\}\\right\\rVert^\{2\}\.\(19\)The gradient norm is therefore a cheap loss\-sensitivity screen, but it is not a complete layer\-selection rule: activation scale, dimensionality, curvature, and downstream policy effects differ across sites\.

#### Fixed\-direction calibration\.

LetFlF\_\{l\}be the activation\-space empirical Fisher and writeπα,l\\pi\_\{\\alpha,l\}for the policy induced byδl=α​ql\\delta\_\{l\}=\\alpha q\_\{l\}\. A local quadratic approximation to its action KL gives

DKL​\(π,πα,l\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(\\pi,\\pi\_\{\\alpha,l\}\)≈12​α2​ql⊤​Fl​ql,\\displaystyle\\approx\\tfrac\{1\}\{2\}\\alpha^\{2\}q\_\{l\}^\{\\top\}F\_\{l\}q\_\{l\},\(20\)αl\\displaystyle\\alpha\_\{l\}=2​εql⊤​Fl​ql,\\displaystyle=\\sqrt\{\\frac\{2\\varepsilon\}\{q\_\{l\}^\{\\top\}F\_\{l\}q\_\{l\}\}\},where the second line sets the approximation equal to the policy\-change budgetε\\varepsilon\. This is the fixed\-direction calibration used by PGS\.

#### Policy\-aware site sensitivity\.

A more general policy\-aware site diagnostic also allows the intervention direction to change\. The KL\-constrained local problem and its solution are

maxδl⁡ql⊤​δl\\displaystyle\\max\_\{\\delta\_\{l\}\}\\ q\_\{l\}^\{\\top\}\\delta\_\{l\}s\.t\.12​δl⊤​Fl​δl≤ε,\\displaystyle\\text\{s\.t\.\}\\quad\\tfrac\{1\}\{2\}\\delta\_\{l\}^\{\\top\}F\_\{l\}\\delta\_\{l\}\\leq\\varepsilon,\(21\)δl∗\\displaystyle\\delta\_\{l\}^\{\*\}=2​εql⊤​Fl−1​ql​Fl−1​ql\.\\displaystyle=\\sqrt\{\\frac\{2\\varepsilon\}\{q\_\{l\}^\{\\top\}F\_\{l\}^\{\-1\}q\_\{l\}\}\}F\_\{l\}^\{\-1\}q\_\{l\}\.Thusql⊤​Fl−1​qlq\_\{l\}^\{\\top\}F\_\{l\}^\{\-1\}q\_\{l\}is the predicted objective gain per local policy\-change budget\. Both site\-sensitivity criteria are diagnostics only: the intervention site and finite strength are selected on validation data, and the final action KL and behavioral effect are measured on held\-out data\.

## Appendix EReproducibility

### E\.1Code and Data

Reusable implementations, experiment entrypoints, and frozen configurations are undersrc/,scripts/, andconfigs/\. The configurations specify the model or environment, data split, intervention and calibration settings, and all random, fit, and partition seeds\. For example, the chess experiments use fit seeds 42–46 with a fixed partition seed \(20260723\); comparisons share the corresponding splits and evaluation seeds\. Entry points write per\-seed diagnostics and machine\-readable result rows, from which the paper tables are generated\.

### E\.2Software

Experiments use Python 3\.11 with dependencies managed byuv\(Astral Software Inc[2025](https://arxiv.org/html/2607.27574#bib.bib197)\)\. The numerical, configuration, and plotting stack uses NumPy, Hydra, and Matplotlib\(Harriset al\.[2020](https://arxiv.org/html/2607.27574#bib.bib213); Yadan[2019](https://arxiv.org/html/2607.27574#bib.bib222); Hunter[2007](https://arxiv.org/html/2607.27574#bib.bib212)\)\. Neural\-policy experiments use PyTorch\(Anselet al\.[2024](https://arxiv.org/html/2607.27574#bib.bib221)\); chess experiments uselczerolens\(Poupart[2026](https://arxiv.org/html/2607.27574#bib.bib199)\); football experiments use Google Research Football\(Kurachet al\.[2020](https://arxiv.org/html/2607.27574#bib.bib195)\)\.

### E\.3Hardware

The GPU launches use one NVIDIA Tesla V100 GPU, 10 CPU cores, and 40 GB of host memory per GPU job\. Lightweight diagnostic workflows are run locally\.