基于激活空间引导的可验证奖励强化学习探索方法

# Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards
Source: [https://arxiv.org/html/2610.27012](https://arxiv.org/html/2610.27012)
Jinhao JingAffiliation:The Chinese University of Hong Kong, ShenzhenXi ChengYuwang Wang††thanks:Corresponding author:wang\-yuwang@tsinghua\.edu\.cn\.Affiliation:Kunlun Inc\. Tsinghua University

###### Abstract

Reinforcement learning with verifiable rewards \(RLVR\) trains a language model by sampling responses, scoring them with an automatic verifier, and updating the policy from the resulting rewards\. The policy therefore receives direct learning signals only from trajectories that appear in its rollouts\. Yet repeated sampling from the same policy can concentrate on similar reasoning paths, leaving useful alternatives unexplored under a fixed rollout budget\. We introduce Activation\-Space Exploration \(ASE\), an online method for diversifying training rollouts\. By clustering activation differences between correct and incorrect responses to the same problem, ASE builds a structured bank of latent exploration vectors\. Different vectors are then applied to subsequent rollouts to explore alternative reasoning paths\. On Qwen3\-1\.7B\-Base and Qwen3\-8B\-Base trained on DAPO\-Math\-17k, ASE improves average Mean@8 across six mathematical reasoning benchmarks over GRPO by 4\.87 and 1\.87 percentage points, respectively, under the same rollout budget\.

## 1Introduction

Reinforcement learning improves a model by repeatedly sampling behaviors, evaluating their outcomes with rewards, and increasing the probability of behaviors that receive higher rewards\([Sutton and Barto, 2018](https://arxiv.org/html/2610.27012#bib.bib1)\)\. Reinforcement learning with verifiable rewards \(RLVR\) makes this process scalable by using automated verifiers to assign outcome\-based rewards to generated responses, such as mathematical solutions and programs\([Le et al\., 2022](https://arxiv.org/html/2610.27012#bib.bib2);[Shao et al\., 2024](https://arxiv.org/html/2610.27012#bib.bib25);[Yu et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib33)\)\. A fundamental constraint, however, is that the policy can only learn from behaviors that appear in its sampled rollouts\([Shao et al\., 2024](https://arxiv.org/html/2610.27012#bib.bib25);[Yue et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib34)\)\. Consequently, policy optimization is shaped bywhich trajectories the rollout procedure makes available for learning, a dependence that creates an exploration bottleneck under a finite rollout budget\. Standard GRPO independently samples multiple trajectories from the same current policy\([Shao et al\., 2024](https://arxiv.org/html/2610.27012#bib.bib25)\)\. Increasing the number of rollouts provides more Monte Carlo samples, but leaves the underlying proposal distribution unchanged\. As the policy concentrates probability mass on a limited set of reasoning patterns, multiple samples may repeatedly visit similar solution modes, reproduce the same intermediate steps, or make correlated errors\. Useful alternatives that occupy low\-probability regions of the policy can therefore remain rarely observed and receive little direct reward feedback\. The bottleneck is thus not merely how many trajectories are sampled, buthow a fixed rollout budget is distributed across the policy’s reasoning space\.

Several approaches seek to use a finite RLVR rollout budget more effectively\. Adaptive sampling allocates more rollouts to difficult problems\([Liao et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib36)\), while data filtering and curricula prioritize more informative training problems\([Yu et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib33);[Chen et al\., 2025b](https://arxiv.org/html/2610.27012#bib.bib38)\)\. These methods change where sampling effort is spent, rather than how trajectories for the same problem are generated\. Adjusting decoding temperature increases token\-level variation\([Dang et al\., 2026](https://arxiv.org/html/2610.27012#bib.bib37)\), but does not specify which alternative reasoning patterns to explore\. Replay reuses previously observed trajectories\([Li et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib22)\), while novelty rewards and high\-entropy prefix regeneration diversify generation around observed paths\([Hu et al\., 2026](https://arxiv.org/html/2610.27012#bib.bib20)\)\. Together, these methods improve how rollouts are allocated, change how randomly tokens are drawn, and reuse previously generated trajectories\. They leave the model’s internal representations largely unexplored as a way to construct different rollout proposals for the same prompt\.

We investigate activation\-space exploration, where temporary interventions on hidden representations are used to construct multiple rollout proposal distributions while leaving both the problem context and model parameters unchanged\. Activation steering has shown that directional interventions on Transformer representations can systematically alter model behavior and generation trajectories\([Beaglehole et al\., 2026](https://arxiv.org/html/2610.27012#bib.bib6);[Chen et al\., 2025a](https://arxiv.org/html/2610.27012#bib.bib7)\)\. Rather than using such interventions as inference\-time behavior controllers, we ask whether they can serve as a training\-time exploration mechanism: different latent directions can perturb generation toward different regions of the policy’s reasoning space, exposing the optimizer to useful trajectories that ordinary sampling may rarely encounter\.

Based on this idea, we proposeActivation\-Space Exploration \(ASE\), an online exploration mechanism for RLVR that learns where to explore from the policy’s own verifier results\. For each problem, successful and unsuccessful trajectories generated during training provide within\-problem contrasts that isolate activation differences associated with task outcome while controlling for problem content\. We find that the activation contrasts exhibit heterogeneous structure; we therefore cluster and aggregate them into multiple exploration directions maintained in an online vector bank\. During subsequent rollout generation, different trajectories for the same problem are routed through different latent directions, producing a collection of steering\-conditioned proposal distributions rather than repeatedly sampling from a single one \(Figure[1](https://arxiv.org/html/2610.27012#S1.F1)\)\. Stochastic decoding is retained within every route, so ASE complements rather than replaces ordinary token sampling\. Verifier rewards from the steered rollouts drive both subsequent exploration updates and optimization of the base policy, leaving evaluation and deployment intervention\-free\.

![Refer to caption](https://arxiv.org/html/2610.27012v1/overview.png)Figure 1:Motivation and high\-level overview of activation\-space exploration\. Standard GRPO samples all responses from one proposal distribution, whereas ASE uses outcome feedback to maintain multiple activation\-space directions, routes responses through different proposals, and updates the actor and vector bank online\.We evaluate ASE on Qwen3\-1\.7B\-Base\([Qwen Team, 2025](https://arxiv.org/html/2610.27012#bib.bib26)\)and Qwen3\-8B\-Base\([Qwen Team, 2025](https://arxiv.org/html/2610.27012#bib.bib26)\)using DAPO\-Math\-17k\([Yu et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib33)\)under the same rollout budget as standard GRPO\. All activation interventions are disabled during evaluation\. Across six mathematical reasoning benchmarks, ASE improves average Mean@8 over GRPO by4\.87 percentage points on Qwen3\-1\.7B\-Base and 1\.87 points on Qwen3\-8B\-Base, while also transferring to out\-of\-distribution reasoning benchmarks\. These results show that activation\-space exploration during training can improve the learned policy across model scales and benchmark distributions under a fixed rollout budget\.

## 2Related Work

#### Reinforcement learning with verifiable rewards\.

RLVR is now a major paradigm for improving mathematical and code reasoning in language models\.[Shao et al\. \(2024\)](https://arxiv.org/html/2610.27012#bib.bib25)introduced GRPO, which constructs policy\-gradient signals from relative rewards among multiple trajectories for the same problem; subsequent work studied the stability and efficiency of scaling this formulation\([Yu et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib33)\)\. RLVR gains depend strongly on trajectories collected by the current policy\. Policy optimization may mostly redistribute probability mass toward high\-reward paths already available to the base model and can reduce solution\-space coverage under large\-scale sampling\([Yue et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib34)\); policy entropy can also decrease continuously and eventually limit further exploration\([Cui et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib16)\)\. At the token level, a small number of high\-entropy branching positions have a disproportionate effect on reasoning paths\([Wang et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib32)\)\. Recent methods encourage flatter distributions and answer diversity in the objective\([Li et al\., 2026](https://arxiv.org/html/2610.27012#bib.bib23)\), or combine trajectory novelty rewards with high\-entropy prefix regeneration to discover different correct solutions\([Hu et al\., 2026](https://arxiv.org/html/2610.27012#bib.bib20)\)\. RePO retrieves historical off\-policy trajectories from a replay buffer and estimates their advantages separately from current on\-policy rollouts\([Li et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib22)\); GRESO uses reward dynamics across epochs to probabilistically skip prompts expected to remain zero\-variance\([Zheng et al\., 2025b](https://arxiv.org/html/2610.27012#bib.bib35)\)\. These methods improve exploration through objectives, reward shaping, explicit trajectory operations, or data scheduling\. We instead use historical rollout outcomes to construct multiple directional proposals in activation space, directly changing the sampling distribution of later training trajectories without an additional diversity reward or a modified GRPO objective\.

#### Activation steering\.

Activation steering builds on the observation that high\-level behaviors have controllable directions in language\-model representations\. Representation engineering and inference\-time intervention show that identifying and modifying intermediate representations can change truthfulness, safety, and other behaviors\([Zou et al\., 2023](https://arxiv.org/html/2610.27012#bib.bib39);[Li et al\., 2023](https://arxiv.org/html/2610.27012#bib.bib21)\)\. More systematic studies have verified transferable and compositional linear concept representations across hundreds of concepts in language, vision\-language, and reasoning models, and demonstrated their use for steering and monitoring\([Beaglehole et al\., 2026](https://arxiv.org/html/2610.27012#bib.bib6)\)\. Activation Addition \(ActAdd\) constructs a direction from activation differences of contrastive prompts\([Turner et al\., 2023](https://arxiv.org/html/2610.27012#bib.bib30)\); Contrastive Activation Addition \(CAA\) pairs examples with opposite behavior labels, averages residual\-stream differences, and adds the result during generation\([Rimsky et al\., 2024](https://arxiv.org/html/2610.27012#bib.bib24)\)\. Later work extends this idea to instruction following\([Stolfo et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib29)\)and reasoning behaviors such as uncertainty, backtracking, and hypothesis testing\([Venhoff et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib31)\)\. Other methods automatically construct steering vectors from labeled data\([Cui and Chen, 2026](https://arxiv.org/html/2610.27012#bib.bib17)\), or optimize a small number of trainable vectors with RL so that they carry reasoning gains\([Sinii et al\., 2026](https://arxiv.org/html/2610.27012#bib.bib28)\)\. These methods generally treat a steering vector as a behavior controller applied at inference time and rely on predefined attributes, externally constructed contrast sets, or optimization of the vector itself\. We reposition activation steering as a temporary training\-time exploration mechanism: directions are estimated online from success and failure trajectories of the current policy, several directions form heterogeneous rollout proposals, and GRPO absorbs useful behaviors into the base policy parameters\. The vector bank is discarded after training\.

## 3Method

### 3\.1Overview

Letx∼𝒟x\\sim\\mathcal\{D\}be a problem and letπθ\\pi\_\{\\theta\}be a language\-model policy\. For a fixedxx, standard GRPO independently samplesNNtrajectories:

yi\(t\)∼πθ\(t\)\(⋅∣x\),i=1,…,N\.y\_\{i\}^\{\(t\)\}\\sim\\pi\_\{\\theta^\{\(t\)\}\}\(\\cdot\\mid x\),\\qquad i=1,\\ldots,N\.\(1\)With a fixed policy and sampling configuration, all trajectories are generated under the same proposal distribution\. IncreasingNNgives more Monte Carlo samples while leaving the proposal distribution unchanged\. We use historical rollout trajectories to construct structured activation\-space exploration directions\.

These exploration directions form a steering vector bank that is updated across iterations\. At iterationtt,

𝒱\(t\)=\{𝐯k\(t\)∈ℝd\}k=1K,∥𝐯k\(t\)∥2=1\.\\mathcal\{V\}^\{\(t\)\}=\\\{\\mathbf\{v\}\_\{k\}^\{\(t\)\}\\in\\mathbb\{R\}^\{d\}\\\}\_\{k=1\}^\{K\},\\qquad\\\|\\mathbf\{v\}\_\{k\}^\{\(t\)\}\\\|\_\{2\}=1\.\(2\)Hereddis the hidden dimension of the target Transformer block andKKis the number of exploration routes\. Each vector is a smoothed direction obtained from mean activation differences between correct and incorrect trajectories in a cluster\. The vector bank is a non\-parametric training state independent of actor parameters and is estimated directly from historical rollouts\.

During rollout, trajectories for the same problem are assigned different routes\. For routesi∈\{1,…,K\}s\_\{i\}\\in\\\{1,\\ldots,K\\\}, we addα​𝐯si\(t\)\\alpha\\mathbf\{v\}\_\{s\_\{i\}\}^\{\(t\)\}to the output of the target Transformer block during generation\. The scalarα\\alphacontrols the intervention strength\. We also include a clean route with no activation intervention, represented by𝐯0\(t\)=𝟎\\mathbf\{v\}\_\{0\}^\{\(t\)\}=\\mathbf\{0\}\. This changes hidden states and subsequent token distributions, yielding a route\-specific proposal distribution:

yi\(t\)∼\{πθ\(t\)\(⋅∣x\),si=0,π~θ\(t\),𝐯si\(t\)\(⋅∣x\),si∈\{1,…,K\}\.y\_\{i\}^\{\(t\)\}\\sim\\begin\{cases\}\\pi\_\{\\theta^\{\(t\)\}\}\(\\cdot\\mid x\),&s\_\{i\}=0,\\\\ \\widetilde\{\\pi\}\_\{\\theta^\{\(t\)\},\\mathbf\{v\}\_\{s\_\{i\}\}^\{\(t\)\}\}\(\\cdot\\mid x\),&s\_\{i\}\\in\\\{1,\\ldots,K\\\}\.\\end\{cases\}\(3\)Token sampling remains stochastic within each route, and the main configuration allocates the group budget uniformly across the clean route and theKKsteering\-guided proposals\. If𝒱\(t\)=∅\\mathcal\{V\}^\{\(t\)\}=\\emptyset, the rollout distribution reduces to the unsteered policyπθ\(t\)\(⋅∣x\)\\pi\_\{\\theta^\{\(t\)\}\}\(\\cdot\\mid x\)\.

The verifier supplies a rewardri\(t\)r\_\{i\}^\{\(t\)\}after generation\. GRPO converts group rewards into relative advantages and updates the base policy with these routed trajectories\. Steering changes only the proposal distribution during rollout, allowing training to discover candidates rarely covered by ordinary sampling\. GRPO then increases the probability of useful behaviors under the base policy, making the exploratory experience persistent\. Validation and deployment disable steering, so the resulting policy can reason independently\.

The same verifier\-scored trajectories update future exploration\. We construct positive–negative pairs to estimate𝒱\(t\+1\)\\mathcal\{V\}^\{\(t\+1\)\}; a newly estimated bank becomes active only after actor optimization and weight synchronization\. The resulting loop is structured exploration, outcome evaluation, policy learning, and exploration update \(Figure[2](https://arxiv.org/html/2610.27012#S3.F2)\)\. Section[3\.2](https://arxiv.org/html/2610.27012#S3.SS2)describes vector\-bank estimation and Section[3\.3](https://arxiv.org/html/2610.27012#S3.SS3)describes steering\-guided rollout optimization\.

![Refer to caption](https://arxiv.org/html/2610.27012v1/pipeline.png)Figure 2:ASE pipeline\. We replay successful and unsuccessful responses to read out hidden activations, cluster their pairwise differences to form exploration directions, match and smooth the directions across updates, and inject one selected direction into the residual stream for each routed response\.
### 3\.2Online Exploration Vector Bank

#### Within\-problem positive–negative pairing\.

For each problem, we compare trajectories with positive and negative rewards\. Positive trajectories are successful samples from the current policy, while negative trajectories provide unsuccessful references\. Let𝒫=\{i:ri=1\}\\mathcal\{P\}=\\\{i:r\_\{i\}=1\\\}and𝒩=\{i:ri=0\}\\mathcal\{N\}=\\\{i:r\_\{i\}=0\\\}\. Only groups containing both reward types provide a within\-problem contrast\. For each such group, we randomly select and pairnpair=min⁡\(\|𝒫\|,\|𝒩\|\)n\_\{\\mathrm\{pair\}\}=\\min\(\|\\mathcal\{P\}\|,\|\\mathcal\{N\}\|\)trajectories from the two sides\. Groups whose rewards are all one or all zero do not contribute to the current steering state\.

#### Activation readout\.

We replay each selected full prompt–response sequence with the current actor without steering\. Let𝐡i=ϕℓ,p​\(x,yi\)∈ℝd\\mathbf\{h\}\_\{i\}=\\phi\_\{\\ell,p\}\(x,y\_\{i\}\)\\in\\mathbb\{R\}^\{d\}be the activation selected by position functionppfrom the output of Transformer blockℓ\\ell\. Each within\-problem pair gives a raw activation difference𝐝j=𝐡j\+−𝐡j−\\mathbf\{d\}\_\{j\}=\\mathbf\{h\}^\{\+\}\_\{j\}\-\\mathbf\{h\}^\{\-\}\_\{j\}forj=1,…,npairj=1,\\ldots,n\_\{\\mathrm\{pair\}\}\. Because the paired trajectories share the same prompt and use the same readout position rulepp, this difference removes much of the variation caused by problem content and difficulty and provides a basis for structured exploration directions\.

Paired activations accumulate across iterations in a bufferℬ\\mathcal\{B\}\. LetBminB\_\{\\min\}be the minimum number of pairs required for an update,Δ\\Deltathe minimum interval between updates, andtlastt\_\{\\mathrm\{last\}\}the iteration index of the most recent vector\-bank update\. When\|ℬ\|≥Bmin\|\\mathcal\{B\}\|\\geq B\_\{\\min\}andt−tlast≥Δt\-t\_\{\\mathrm\{last\}\}\\geq\\Delta, we re\-estimate the vector bank from the full buffer and then clear it\. The readout layer, position function,BminB\_\{\\min\}, andΔ\\Deltaare implementation hyperparameters\.

#### Difference clustering\.

A single global mean difference would compress diverse exploration patterns into one direction\. These patterns may reflect variation in early strategy choice, the sequencing of intermediate steps, or later checking and revision\. We therefore cluster in a dimension\-reduced pair\-difference space\.

Suppose the buffer containsMMpairs\{\(𝐡b\+,𝐡b−\)\}b=1M\\\{\(\\mathbf\{h\}\_\{b\}^\{\+\},\\mathbf\{h\}\_\{b\}^\{\-\}\)\\\}\_\{b=1\}^\{M\}collected across problems\. To stabilize clustering when the number of pairs is limited relative to the hidden dimension, we fit a PCA projection𝐐∈ℝdPCA×d\\mathbf\{Q\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{PCA\}\}\\times d\}on the pooled positive and negative activations and compute each pair’s difference in this shared subspace as𝜹b=𝐐𝐡b\+−𝐐𝐡b−\\bm\{\\delta\}\_\{b\}=\\mathbf\{Q\}\\mathbf\{h\}\_\{b\}^\{\+\}\-\\mathbf\{Q\}\\mathbf\{h\}\_\{b\}^\{\-\}\. HeredPCAd\_\{\\mathrm\{PCA\}\}denotes the number of retained principal components\. K\-means partitions\{𝜹b\}b=1M\\\{\\bm\{\\delta\}\_\{b\}\\\}\_\{b=1\}^\{M\}intoKKclusters\{𝒞k\(t\+1\)\}k=1K\\\{\\mathcal\{C\}\_\{k\}^\{\(t\+1\)\}\\\}\_\{k=1\}^\{K\}\. The set𝒞k\(t\+1\)\\mathcal\{C\}\_\{k\}^\{\(t\+1\)\}contains the indices of all pairs assigned to clusterkk\. PCA only defines the clustering space\. The final direction is the within\-cluster mean difference in the original hidden space:

𝐮^k\(t\+1\)=1\|𝒞k\(t\+1\)\|​∑b∈𝒞k\(t\+1\)\(𝐡b\+−𝐡b−\)\.\\widehat\{\\mathbf\{u\}\}\_\{k\}^\{\(t\+1\)\}=\\frac\{1\}\{\|\\mathcal\{C\}\_\{k\}^\{\(t\+1\)\}\|\}\\sum\_\{b\\in\\mathcal\{C\}\_\{k\}^\{\(t\+1\)\}\}\(\\mathbf\{h\}\_\{b\}^\{\+\}\-\\mathbf\{h\}\_\{b\}^\{\-\}\)\.\(4\)The clusters construct multiple local rollout proposals; we do not assume that they correspond to discrete, stable, or human\-namable semantics\.

#### Cross\-iteration stabilization\.

Let𝐮k\(t\)\\mathbf\{u\}\_\{k\}^\{\(t\)\}denote the unnormalized, temporally smoothed direction, with𝐯k\(t\)=𝐮k\(t\)/‖𝐮k\(t\)‖2\\mathbf\{v\}\_\{k\}^\{\(t\)\}=\\mathbf\{u\}\_\{k\}^\{\(t\)\}/\\\|\\mathbf\{u\}\_\{k\}^\{\(t\)\}\\\|\_\{2\}\. A finite buffer is affected by rollout noise, so we use an EMA with coefficientβ∈\[0,1\)\\beta\\in\[0,1\)to smooth updates\. Since K\-means cluster indices have no fixed meaning across updates, we first align the new candidate directions with the existing bank by solving

P\(t\+1\)=arg⁡max⁡∑k=1KP∈𝔖K⁡cos⁡\(𝐮k\(t\),𝐮^P⁡\(k\)\(t\+1\)\),P^\{\(t\+1\)\}=\\arg\\max\_\{P\\in\\mathfrak\{S\}\_\{K\}\}\\sum\_\{k=1\}^\{K\}\\cos\(\\mathbf\{u\}\_\{k\}^\{\(t\)\},\\widehat\{\\mathbf\{u\}\}\_\{P\(k\)\}^\{\(t\+1\)\}\),\(5\)where𝔖K\\mathfrak\{S\}\_\{K\}is the set of permutations\. We use the Hungarian algorithm, then update

𝐮k\(t\+1\)=β​𝐮k\(t\)\+\(1−β\)​𝐮^P\(t\+1\)​\(k\)\(t\+1\)\.\\mathbf\{u\}\_\{k\}^\{\(t\+1\)\}=\\beta\\mathbf\{u\}\_\{k\}^\{\(t\)\}\+\(1\-\\beta\)\\widehat\{\\mathbf\{u\}\}\_\{P^\{\(t\+1\)\}\(k\)\}^\{\(t\+1\)\}\.\(6\)For the first update, we set𝐮k\(t\+1\)=𝐮^k\(t\+1\)\\mathbf\{u\}\_\{k\}^\{\(t\+1\)\}=\\widehat\{\\mathbf\{u\}\}\_\{k\}^\{\(t\+1\)\}\. Finally, we normalize𝐮k\(t\+1\)\\mathbf\{u\}\_\{k\}^\{\(t\+1\)\}to unitL2L\_\{2\}norm to obtain𝐯k\(t\+1\)\\mathbf\{v\}\_\{k\}^\{\(t\+1\)\}\. The new bank is staged until actor optimization and rollout weight synchronization finish\.

### 3\.3Steering\-Guided Rollout Optimization

#### Routed activation steering\.

When𝒱\(t\)≠∅\\mathcal\{V\}^\{\(t\)\}\\neq\\emptyset, we assign routessi∈\{0,…,K\}s\_\{i\}\\in\\\{0,\\ldots,K\\\}to trajectories for each problem, wheresi=0s\_\{i\}=0is the clean route\. Our main method uses allK\+1K\+1routes uniformly within each group\. For any target\-layer output𝐡\(ℓ\)\\mathbf\{h\}^\{\(\\ell\)\}on a trajectory assigned to a steering routesi∈\{1,…,K\}s\_\{i\}\\in\\\{1,\\ldots,K\\\}, the intervention is

𝐡~\(ℓ\)=𝐡\(ℓ\)\+α​𝐯si\(t\)\.\\widetilde\{\\mathbf\{h\}\}^\{\(\\ell\)\}=\\mathbf\{h\}^\{\(\\ell\)\}\+\\alpha\\mathbf\{v\}\_\{s\_\{i\}\}^\{\(t\)\}\.\(7\)For the clean route, the output is unchanged\. The intervention is applied to every forward pass of the target layer, including prompt prefill and autoregressive decoding\. Since each vector has unitL2L\_\{2\}norm,α\\alphadirectly controls the injected residual\. Steering is executed by the rollout engine with fixed actor parameters and is outside backpropagation\.

#### GRPO update\.

For theNNrollouts of a fixed problemxx, letμ\\muandσ\\sigmabe the mean and sample standard deviation of their verifier rewards\. The group\-relative advantage isAi=\(ri−μ\)/\(σ\+ϵ\)A\_\{i\}=\(r\_\{i\}\-\\mu\)/\(\\sigma\+\\epsilon\), whereϵ\>0\\epsilon\>0stabilizes normalization\. We update the base policy using trajectories from both clean and steered rollout routes, and apply a bounded token\-level importance weight in the clipped GRPO objective\. For each response token,

ρi,τ​\(θ\)=πθ​\(yi,τ∣x,yi,<τ\)πθ\(t\)​\(yi,τ∣x,yi,<τ\)\.\\rho\_\{i,\\tau\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,\\tau\}\\mid x,y\_\{i,<\\tau\}\)\}\{\\pi\_\{\\theta^\{\(t\)\}\}\(y\_\{i,\\tau\}\\mid x,y\_\{i,<\\tau\}\)\}\.\(8\)Forsi∈\{0,…,K\}s\_\{i\}\\in\\\{0,\\ldots,K\\\}, the truncated importance ratio is

wi,τ=min⁡\(cIS,πθ\(t\)​\(yi,τ∣x,yi,<τ\)π~θ\(t\),𝐯si\(t\)​\(yi,τ∣x,yi,<τ\)\)\.w\_\{i,\\tau\}=\\min\\\!\\left\(c\_\{\\mathrm\{IS\}\},\\frac\{\\pi\_\{\\theta^\{\(t\)\}\}\(y\_\{i,\\tau\}\\mid x,y\_\{i,<\\tau\}\)\}\{\\widetilde\{\\pi\}\_\{\\theta^\{\(t\)\},\\mathbf\{v\}\_\{s\_\{i\}\}^\{\(t\)\}\}\(y\_\{i,\\tau\}\\mid x,y\_\{i,<\\tau\}\)\}\\right\)\.\(9\)HerecIS\>0c\_\{\\mathrm\{IS\}\}\>0caps the importance weight\. This weight is computed separately at each response position\. The token\-level objective is

ℒ⁡\(θ\)=−𝔼x,i,τ​\[wi,τ​min⁡\(ρi,τ​\(θ\)​Ai,clip⁡\(ρi,τ​\(θ\),1−ε,1\+ε\)​Ai\)\]\.\\mathcal\{L\}\(\\theta\)=\-\\mathbb\{E\}\_\{x,i,\\tau\}\\\!\\left\[w\_\{i,\\tau\}\\min\\left\(\\rho\_\{i,\\tau\}\(\\theta\)A\_\{i\},\\operatorname\{clip\}\(\\rho\_\{i,\\tau\}\(\\theta\),1\-\\varepsilon,1\+\\varepsilon\)A\_\{i\}\\right\)\\right\]\.\(10\)The expectation averages over generated response tokens, andε\>0\\varepsilon\>0sets the PPO clipping range\. Minimizing this objective updates the base policy fromθ\(t\)\\theta^\{\(t\)\}toθ\(t\+1\)\\theta^\{\(t\+1\)\}\. The verifier and advantage estimator follow GRPO\. The bounded importance weight adjusts the contribution of each sampled token according to its likelihood under the base policy and its rollout route\. The complete training procedure is given in Algorithm[1](https://arxiv.org/html/2610.27012#algorithm1)\.

## 4Experiments

### 4\.1Experimental Setup

#### Models and datasets\.

We train Qwen3\-1\.7B\-Base and Qwen3\-8B\-Base\([Qwen Team, 2025](https://arxiv.org/html/2610.27012#bib.bib26)\)with DAPO\-Math\-17k\([Yu et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib33)\)\. We report six mathematical test benchmarks: MATH500\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.27012#bib.bib8);[Lightman et al\., 2023](https://arxiv.org/html/2610.27012#bib.bib9)\), Minerva\([Lewkowycz et al\., 2022](https://arxiv.org/html/2610.27012#bib.bib11)\), AIME24\([Zhang and Math\-AI Team, 2024](https://arxiv.org/html/2610.27012#bib.bib3)\), AIME25\([Zhang and Math\-AI Team, 2025](https://arxiv.org/html/2610.27012#bib.bib4)\), AMC24\([ExtraordinaryLab, 2025](https://arxiv.org/html/2610.27012#bib.bib5)\), and OlympiadBench\([He et al\., 2024](https://arxiv.org/html/2610.27012#bib.bib12)\); and three out\-of\-distribution benchmarks: MMLU\-Pro\([Wang et al\., 2024](https://arxiv.org/html/2610.27012#bib.bib13)\), GPQA\-Diamond\([Rein et al\., 2024](https://arxiv.org/html/2610.27012#bib.bib14)\), and BBH\([Suzgun et al\., 2023](https://arxiv.org/html/2610.27012#bib.bib15)\)\.

#### Baselines\.

We compare with standard GRPO and four representative baselines\. DAPO\([Yu et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib33)\)dynamically identifies and discards zero\-variance prompts; Dr\.GRPO\([Liu et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib18)\)redesigns advantages and loss aggregation to address length bias; GSPO\([Zheng et al\., 2025a](https://arxiv.org/html/2610.27012#bib.bib19)\)performs importance weighting, clipping, and optimization at the sequence level; and RePO\([Li et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib22)\)reuses complete historical trajectories from a replay buffer\. We follow method\-specific configurations from the corresponding papers or public implementations\.

#### Implementation details\.

All methods use four epochs, a training and actor mini\-batch size of 64, and eight rollouts per problem\. The actor is optimized with AdamW \(learning rate1×10−61\\times 10^\{\-6\}, weight decay 0\.01\), a constant learning\-rate schedule, and no warmup; we use neither KL loss nor entropy regularization\. Training rollouts use temperature 1\.0, top\-pp1\.0, top\-k=−1k=\-1, and maximum prompt/response lengths of 1024/4096, while final evaluation uses temperature 0\.6, top\-pp0\.95, top\-k=0k=0, and maximum response length 4096\. For ASE, we useK=3K=3steering clusters plus a clean route, steering strengthα=0\.5\\alpha=0\.5, and EMA coefficientβ=0\.5\\beta=0\.5\. Training uses verl\([Sheng et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib27)\), vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.27012#bib.bib10)\), BF16, and FlashAttention\-2\([Dao, 2024](https://arxiv.org/html/2610.27012#bib.bib40)\)on four NVIDIA A800 GPUs for the 1\.7B model and eight for the 8B model\. Math\-Verify111[https://github\.com/huggingface/Math\-Verify](https://github.com/huggingface/Math-Verify)checks final answers\. Shared training and evaluation settings are given in Appendix[C](https://arxiv.org/html/2610.27012#A3); checkpoint\-selection and remaining method\-specific details are provided in Appendices[B](https://arxiv.org/html/2610.27012#A2)and[C](https://arxiv.org/html/2610.27012#A3)\.

### 4\.2Main Results

Tables[1](https://arxiv.org/html/2610.27012#S4.T1)and[2](https://arxiv.org/html/2610.27012#S4.T2)present the main results for Qwen3\-8B\-Base and Qwen3\-1\.7B\-Base, respectively\. Each table cell reports Mean@8 / Pass@8 in percent\. Mean@8 is the average accuracy over eight sampled answers; Pass@8 is the fraction of problems with at least one correct answer\. Reported scores are averaged over three runs\. Math Avg\. and OOD Avg\. are the unweighted means of benchmark\-level scores across the six mathematical and three out\-of\-distribution benchmarks, respectively, computed separately for Mean@8 and Pass@8\.

Table 1:Main results for Qwen3\-8B\-Base trained on DAPO\-Math\-17k\. Scores are reported as Mean@8 / Pass@8 \(in percent\)\. Reported scores are averaged over three runs\. EachΔ\\Deltarow reports the absolute difference from the corresponding baseline in percentage points\.Mathematical benchmarksOOD benchmarksMethodMATH500MinervaAIME24AIME25AMC24OlympiadMath Avg\.MMLU\-ProGPQA\-DBBHOOD Avg\.GRPO88\.17/94\.8743\.98/55\.7626\.39/48\.8921\.39/37\.7858\.06/82\.2255\.80/70\.1848\.96/64\.9567\.62/85\.6449\.45/79\.8076\.73/91\.1964\.60/85\.54DAPO86\.09/93\.2042\.56/54\.1721\.53/41\.1118\.06/30\.0048\.98/74\.0751\.38/66\.8244\.77/59\.8961\.28/83\.3846\.70/80\.8176\.71/91\.3261\.56/85\.17Dr\.GRPO87\.83/95\.8743\.20/59\.4425\.42/47\.7820\.69/37\.7855\.56/77\.7857\.78/74\.7848\.41/65\.5765\.20/86\.0048\.34/84\.8577\.79/91\.9963\.78/87\.61GSPO88\.05/95\.0043\.01/55\.5125\.00/53\.3322\.50/36\.6758\.33/80\.0056\.71/72\.4048\.94/65\.4964\.94/84\.0847\.47/80\.8175\.74/91\.2062\.72/85\.36RePO86\.09/94\.5341\.57/54\.2922\.64/42\.2218\.61/32\.2253\.06/81\.4853\.34/71\.3645\.89/62\.6961\.21/84\.4043\.96/81\.3167\.92/91\.3557\.69/85\.69ASE\-GRPO89\.31/95\.7343\.70/56\.7428\.19/46\.6721\.67/36\.6762\.78/82\.9659\.32/74\.0950\.83/65\.4866\.63/85\.6851\.01/82\.3276\.26/92\.7464\.63/86\.91Δ\\Deltavs\. GRPO\+1\.14/\+0\.86\-0\.28/\+0\.98\+1\.80/\-2\.22\+0\.28/\-1\.11\+4\.72/\+0\.74\+3\.52/\+3\.91\+1\.87/\+0\.53\-0\.99/\+0\.04\+1\.56/\+2\.52\-0\.47/\+1\.55\+0\.03/\+1\.37Δ\\Deltavs\. DAPO\+3\.22/\+2\.53\+1\.14/\+2\.57\+6\.66/\+5\.56\+3\.61/\+6\.67\+13\.80/\+8\.89\+7\.94/\+7\.27\+6\.06/\+5\.59\+5\.35/\+2\.30\+4\.31/\+1\.51\-0\.45/\+1\.42\+3\.07/\+1\.74Δ\\Deltavs\. Dr\.GRPO\+1\.48/\-0\.14\+0\.50/\-2\.70\+2\.77/\-1\.11\+0\.98/\-1\.11\+7\.22/\+5\.18\+1\.54/\-0\.69\+2\.42/\-0\.09\+1\.43/\-0\.32\+2\.67/\-2\.53\-1\.53/\+0\.75\+0\.85/\-0\.70Δ\\Deltavs\. GSPO\+1\.26/\+0\.73\+0\.69/\+1\.23\+3\.19/\-6\.66\-0\.83/\+0\.00\+4\.45/\+2\.96\+2\.61/\+1\.69\+1\.89/\-0\.01\+1\.69/\+1\.60\+3\.54/\+1\.51\+0\.52/\+1\.54\+1\.91/\+1\.55Δ\\Deltavs\. RePO\+3\.22/\+1\.20\+2\.13/\+2\.45\+5\.55/\+4\.45\+3\.06/\+4\.45\+9\.72/\+1\.48\+5\.98/\+2\.73\+4\.94/\+2\.79\+5\.42/\+1\.28\+7\.05/\+1\.01\+8\.34/\+1\.39\+6\.94/\+1\.22

Table 2:Main results for Qwen3\-1\.7B\-Base trained on DAPO\-Math\-17k\. Scores are reported as Mean@8 / Pass@8 \(in percent\)\. Reported scores are averaged over three runs\. EachΔ\\Deltarow reports the absolute difference from the corresponding baseline in percentage points\.Mathematical benchmarksOOD benchmarksMethodMATH500MinervaAIME24AIME25AMC24OlympiadMath Avg\.MMLU\-ProGPQA\-DBBHOOD Avg\.GRPO65\.59/83\.4727\.37/43\.506\.39/14\.445\.42/22\.2218\.24/40\.7429\.82/50\.5925\.47/42\.5035\.46/69\.0926\.45/79\.1244\.70/84\.1535\.54/77\.45DAPO68\.02/84\.2027\.94/45\.2210\.00/26\.675\.00/18\.8921\.20/43\.7030\.87/50\.4927\.17/44\.8639\.22/69\.2729\.12/76\.6050\.43/85\.2839\.59/77\.05Dr\.GRPO68\.16/84\.5330\.39/45\.5912\.08/23\.332\.92/12\.2221\.39/40\.7431\.43/51\.6827\.73/43\.0240\.54/70\.3330\.30/77\.6151\.99/85\.4340\.94/77\.79GSPO68\.42/84\.8028\.71/44\.369\.58/22\.224\.86/18\.8918\.24/39\.2632\.00/53\.8126\.97/43\.8938\.38/70\.1527\.90/78\.9647\.02/84\.7537\.77/77\.95RePO65\.42/84\.6726\.19/44\.617\.08/22\.224\.31/15\.5617\.78/42\.2229\.60/51\.8825\.06/43\.5336\.15/69\.4526\.54/76\.2644\.36/84\.9635\.68/76\.89ASE\-GRPO71\.15/88\.4029\.78/47\.0610\.83/25\.568\.19/21\.1127\.96/56\.3034\.12/54\.6530\.34/48\.8541\.35/71\.1229\.95/79\.9752\.46/85\.8741\.25/78\.98Δ\\Deltavs\. GRPO\+5\.56/\+4\.93\+2\.41/\+3\.56\+4\.44/\+11\.12\+2\.77/\-1\.11\+9\.72/\+15\.56\+4\.30/\+4\.06\+4\.87/\+6\.35\+5\.89/\+2\.03\+3\.50/\+0\.85\+7\.76/\+1\.72\+5\.71/\+1\.53Δ\\Deltavs\. DAPO\+3\.13/\+4\.20\+1\.84/\+1\.84\+0\.83/\-1\.11\+3\.19/\+2\.22\+6\.76/\+12\.60\+3\.25/\+4\.16\+3\.17/\+3\.99\+2\.13/\+1\.85\+0\.83/\+3\.37\+2\.03/\+0\.59\+1\.66/\+1\.93Δ\\Deltavs\. Dr\.GRPO\+2\.99/\+3\.87\-0\.61/\+1\.47\-1\.25/\+2\.23\+5\.27/\+8\.89\+6\.57/\+15\.56\+2\.69/\+2\.97\+2\.61/\+5\.83\+0\.81/\+0\.79\-0\.35/\+2\.36\+0\.47/\+0\.44\+0\.31/\+1\.19Δ\\Deltavs\. GSPO\+2\.73/\+3\.60\+1\.07/\+2\.70\+1\.25/\+3\.34\+3\.33/\+2\.22\+9\.72/\+17\.04\+2\.12/\+0\.84\+3\.37/\+4\.96\+2\.97/\+0\.97\+2\.05/\+1\.01\+5\.44/\+1\.12\+3\.48/\+1\.03Δ\\Deltavs\. RePO\+5\.73/\+3\.73\+3\.59/\+2\.45\+3\.75/\+3\.34\+3\.88/\+5\.55\+10\.18/\+14\.08\+4\.52/\+2\.77\+5\.28/\+5\.32\+5\.20/\+1\.67\+3\.41/\+3\.71\+8\.10/\+0\.91\+5\.57/\+2\.09

ASE\-GRPO achieves the highest mathematical Mean@8 average on both model sizes\. On Qwen3\-1\.7B\-Base, it reaches 30\.34, compared with 25\.47 for GRPO and 27\.73 for Dr\.GRPO; its mathematical Pass@8 also leads at 48\.85\. On Qwen3\-8B\-Base, it reaches 50\.83, exceeding GRPO by 1\.87 points and GSPO by 1\.89 points\. Improvements are particularly clear on AMC24 and OlympiadBench, while other methods remain stronger on Minerva or AIME24\.

The gains extend beyond the mathematical training domain\. ASE\-GRPO obtains the highest OOD Mean@8 average on both Qwen3\-1\.7B\-Base \(41\.25\) and Qwen3\-8B\-Base \(64\.63\), compared with 35\.54 and 64\.60 for GRPO, respectively\. The 1\.7B model also achieves the highest OOD Pass@8 average, whereas Dr\.GRPO remains higher on this metric for the 8B model\. Together, these results suggest that activation\-space exploration improves the unsteered policy across model scales, with consistent gains in mathematical reasoning and evidence of transfer to out\-of\-distribution tasks\.

Figure[3](https://arxiv.org/html/2610.27012#S4.F3)extends the evaluation to larger sampling budgets\. ASE\-GRPO achieves higher math\-benchmark Pass@kkthan GRPO acrossk=8k=8to128128on both model scales\. This suggests that the learned policy is more effective at discovering correct solutions when sampling multiple candidate responses\.

### 4\.3Dynamics of Training and Vector\-Bank

Figure[4](https://arxiv.org/html/2610.27012#S4.F4)tracks validation performance and rollout behavior on Qwen3\-1\.7B\-Base and Qwen3\-8B\-Base\. Validation Mean@8 rises during training, and ASE\-GRPO reaches higher values than GRPO on both model scales\. Training reward follows a similar trajectory for the two methods, and response lengths remain within a comparable range\. We use pairwise disagreement between answers sampled for the same problem as an answer\-level measure of exploration\. The disagreement gap \(ASE\-GRPO minus GRPO\) is positive overall and generally grows over training on both model sizes, showing that ASE\-GRPO explores a broader range of answers alongside its validation gains without a marked change in training reward or response length\.

Figure[5](https://arxiv.org/html/2610.27012#S4.F5)shows how ASE\-GRPO’s vector bank evolves on Qwen3\-1\.7B\-Base\. Each of the three cluster shares is the fraction of activation pairs assigned to that cluster at a bank update\. The shares remain broadly stable and reasonably balanced throughout training, showing that all three directions continue to receive support from the contrast data\. In the*Vector geometry*panel, temporal alignment is the mean cosine similarity between newly extracted vectors and their matched vectors from the previous bank; the median cosine of 0\.73 shows that successive vectors remain similarly oriented\. Inter\-cluster alignment is the mean pairwise cosine similarity among directions in the updated bank; values near the three\-vector lower bound of−0\.5\-0\.5show that the directions remain well separated\.

Figure 3:Pass@kkaveraged over six math benchmarks \(in percent\)\.
Figure 4:Training dynamics of GRPO and ASE\-GRPO on Qwen3\-1\.7B\-Base and Qwen3\-8B\-Base\. The four columns show validation Mean@8, training reward, response length, and the rollout disagreement gap across training steps\.Figure 5:Vector\-bank dynamics for Qwen3\-1\.7B\-Base\.

### 4\.4Inference\-Time Steering versus Training\-Time Steering

Activation steering can be used in two distinct ways\. An inference\-time intervention changes the output distribution of a fixed model during evaluation, whereas our training\-time intervention diversifies the rollouts used to train the policy\. We compare these two uses of steering to determine whether the main benefit comes from the direct effect of steering a fixed policy or from cumulative gains of online exploration during training\. We compare six conditions: Base, BaseS, GRPO, GRPOS, ASE\-GRPO, and ASE\-GRPOS, whereSdenotes inference\-time intervention\. To assess the potential gains from inference\-time steering, we sweep its parameters separately for each checkpoint; details are reported in Appendix[D](https://arxiv.org/html/2610.27012#A4)\.

Table[3](https://arxiv.org/html/2610.27012#S4.T3)summarizes the comparison\. Inference\-time intervention provides only marginal changes for the untrained and GRPO checkpoints: BaseSchanges Math Mean@8 from 18\.74 to 18\.86 and GRPOSchanges it from 25\.47 to 25\.81\. ASE\-GRPO already outperforms standard GRPO without inference\-time intervention, and applying inference\-time intervention to ASE\-GRPO gives a further 0\.51\-point Math Mean@8 increase and 0\.15\-point OOD Mean@8 increase, while OOD Pass@8 decreases from 78\.98 to 78\.44\. These results indicate that the main gain comes from training\-time exploration, with only modest direct effects from steering a fixed policy\.

### 4\.5Ablating Verifier\-Guided Direction

The vector bank is estimated from within\-problem success–failure contrasts\. This raises a specific attribution question: does verifier feedback matter, or is the gain explained by injecting any additional activation perturbation? We therefore design a strictly matched ablation in which only the construction of the exploration directions varies\. Random directions replace each cluster candidate with an independent random vector of the same norm; shuffled labels randomly swap the positive and negative labels within each pair before clustering and direction estimation; and reversed directions negate the learned vectors while preserving their cluster assignments\. These controls retain the same downstream bank update and routing procedure\.

Table 3:Training\- versus inference\-time steering on Qwen3\-1\.7B\-Base\.Sdenotes inference\-time steering\. Math/OOD Avg\. report Mean@8 / Pass@8 \(%\), averaged over three evaluations\.
Table 4:Direction\-construction ablation on Qwen3\-1\.7B\-Base\. ASE\-GRPO variants differ only in direction construction\. Math/OOD Avg\. report Mean@8 / Pass@8 \(%\), averaged over three evaluations\.

As shown in Table[4](https://arxiv.org/html/2610.27012#S4.T4), ASE\-GRPO outperforms all three ablation variants on both aggregates, reaching 30\.34/48\.85 on Math Avg\. and 41\.25/78\.98 on OOD Avg\. The variants mostly exceed GRPO \(25\.47/42\.50 and 35\.54/77\.45, respectively\), except that shuffled labels has lower OOD Pass@8 \(77\.03 versus 77\.45\)\. These variants retain the clusteredK=3K=3bank, routing, and EMA updates, which may contribute to their gains over GRPO\. An additional control without clustering or EMA yields only a small increase in Math Mean@8 and a decrease in Pass@8 relative to GRPO \(Appendix[D\.2](https://arxiv.org/html/2610.27012#A4.SS2)\), suggesting that arbitrary random steering alone does not provide a consistent benefit\. Together, these results suggest that the shared bank structure and EMA may account for part of the improvement, while verifier\-guided directions provide additional gains within the matched setup\.

### 4\.6Evaluation on Alternative Policy Optimization Formulations

To test whether the benefit of ASE is specific to the standard GRPO formulation, we apply the same training\-time exploration mechanism to Dr\.GRPO and GSPO\. We use Qwen3\-1\.7B\-Base, the same four\-epoch training budget, and eight rollouts per problem for all conditions\. We do not perform a separate parameter search for either alternative formulation and use the preset ASE configuration\. Table[5](https://arxiv.org/html/2610.27012#S4.T5)summarizes the results\. Compared with Dr\.GRPO, ASE\-Dr\.GRPO increases Math Avg\. Mean@8 and Pass@8 by 1\.28 and 2\.18 points, respectively, and OOD Avg\. Mean@8 by 0\.36 points\. Compared with GSPO, ASE\-GSPO increases Math Avg\. Mean@8 and Pass@8 by 1\.49 and 0\.27 points, respectively, and OOD Avg\. Mean@8 and Pass@8 by 2\.08 and 0\.34 points, respectively\. These results indicate that activation\-space exploration also benefits alternative policy\-optimization formulations\.

Table 5:ASE\-Dr\.GRPO and ASE\-GSPO on Qwen3\-1\.7B\-Base\. Entries are Mean@8 / Pass@8 \(%\); Math Avg\. and OOD Avg\. cover six mathematical and three OOD benchmarks, respectively\.
### 4\.7Training\-time overhead\.

Table[6](https://arxiv.org/html/2610.27012#S4.T6)compares latency and throughput for the matched GRPO and ASE\-GRPO runs, averaged over the training trajectory after excluding the first 10 warm\-up steps\. ASE\-GRPO increases token\-normalized rollout latency by only 2\.72% on the 1\.7B model and 2\.51% on the 8B model, while end\-to\-end throughput decreases by 6\.29% and 6\.44%, respectively\. The additional activation replay and vector\-bank maintenance are reflected in the actor\-update latency, but the resulting mean step\-time increases remain modest at 0\.91% and 7\.19%\. Moreover, the vector bank is discarded after training, so deployment incurs no additional inference latency\.

Table 6:Training\-time overhead on matched GRPO and ASE\-GRPO runs\. Throughput uses all prompt and response tokens, rollout latency is normalized by output tokens, and actor\-update latency is normalized by all valid tokens\.

## 5Conclusion

We introduced Activation\-Space Exploration \(ASE\), an online method that uses verifier feedback to construct and update multiple activation\-space directions for training rollouts\. By distributing a fixed rollout budget across these directions, ASE improves mathematical reasoning on two model scales and also benefits Dr\.GRPO and GSPO\. Matched ablations show that success–failure contrasts yield stronger gains than random, label\-shuffled, or reversed directions, while comparisons with inference\-time steering support the value of using these interventions during training\. All steering is removed at evaluation, so the gains persist in the learned base policy without additional inference\-time intervention\. These findings support activation space as a complementary exploration axis for RLVR\.

### AI use statement

In this work, we used generative AI tools to design and provide feedback on research methodology and experiments, assist with the execution of experiments, and assist with translation of parts of the manuscript\. We also used generative AI tools for sourcing and searching for research information, identifying relevant literature, creating and editing software code used in the experiments, and formatting references\. We did not use generative AI tools to generate synthetic datasets, develop theoretical models or conceptual frameworks, formulate mathematical claims, provide critical ingredients for mathematical proofs, assist in writing proofs, propose or refine hypotheses, clean or reformat datasets, support qualitative or thematic data analysis, or interpret the reported results\. All AI\-assisted work was reviewed by the authors\. Experimental procedures, software code, outputs, citations, translations, and reported claims were checked and verified by the authors\. We take responsibility for the final content of this work, including all text, claims, results, and artifacts produced with the aid of generative AI\.

## References

- Beagleholeet al\.\(2026\)D\. Beaglehole, A\. Radhakrishnan, E\. Boix\-Adserà, and M\. BelkinToward universal steering and monitoring of AI models\.Science391\(6787\),pp\. 787–792\.External Links:[Document](https://dx.doi.org/10.1126/science.aea6792),[Link](https://doi.org/10.1126/science.aea6792)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p3.1),[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025a\)R\. Chen, A\. Arditi, H\. Sleight, O\. Evans, and J\. LindseyPersona vectors: monitoring and controlling character traits in language models\.arXiv preprint arXiv:2507\.21509\.External Links:[Link](https://arxiv.org/abs/2507.21509)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p3.1)\.
- Chenet al\.\(2025b\)X\. Chen, J\. Lu, M\. Kim, D\. Zhang, J\. Tang, A\. Piché, N\. Gontier, Y\. Bengio, and E\. KamallooSelf\-evolving curriculum for LLM reasoning\.External Links:2505\.14970,[Link](https://arxiv.org/abs/2505.14970)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p2.1)\.
- Cuiet al\.\(2025\)G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen, Z\. Liu, H\. Peng, L\. Bai, W\. Ouyang, Y\. Cheng, B\. Zhou, and N\. DingThe entropy mechanism of reinforcement learning for reasoning language models\.External Links:2505\.22617,[Document](https://dx.doi.org/10.48550/arXiv.2505.22617),[Link](https://arxiv.org/abs/2505.22617)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1)\.
- Cui and Chen \(2026\)S\. Cui and Z\. ChenPainless activation steering: an automated, lightweight approach for post\-training large language models\.External Links:2509\.22739,[Document](https://dx.doi.org/10.48550/arXiv.2509.22739),[Link](https://arxiv.org/abs/2509.22739)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Danget al\.\(2026\)H\. Dang, C\. Lan, H\. Wan, X\. Zhao, and Y\. LuTemperature as a meta\-policy: adaptive temperature in LLM reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=AoTHU2OmS6)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p2.1)\.
- Dao \(2024\)T\. DaoFlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations,Cited by:[§C\.1](https://arxiv.org/html/2610.27012#A3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px3.p1.1)\.
- ExtraordinaryLab \(2025\)ExtraordinaryLabAMC dataset\.External Links:[Link](https://huggingface.co/datasets/extraordinarylab/amc)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. SunOlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3828–3850\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.211),[Link](https://doi.org/10.18653/v1/2024.acl-long.211)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Vol\.1\.External Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Huet al\.\(2026\)W\. Hu, J\. Jia, Z\. Shu, D\. Dahlmeier, S\. Ng, and B\. K\. H\. LowExTra: exploratory trajectory optimization for language model reinforcement learning\.External Links:2606\.24994,[Document](https://dx.doi.org/10.48550/arXiv.2606.24994),[Link](https://arxiv.org/abs/2606.24994)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p2.1),[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165),[Link](https://doi.org/10.1145/3600006.3613165)Cited by:[§C\.1](https://arxiv.org/html/2610.27012#A3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px3.p1.1)\.
- Leet al\.\(2022\)H\. Le, Y\. Wang, A\. D\. Gotmare, S\. Savarese, and S\. C\. H\. HoiCodeRL: mastering code generation through pretrained models and deep reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 21314–21328\.External Links:[Document](https://dx.doi.org/10.52202/068431-1549),[Link](https://doi.org/10.52202/068431-1549)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p1.1)\.
- Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. MisraSolving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 3843–3857\.External Links:[Document](https://dx.doi.org/10.52202/068431-0278),[Link](https://doi.org/10.52202/068431-0278)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2026\)C\. Li, Y\. Zhuang, C\. Gao, H\. Sun, R\. Qiang, C\. Zhang, and B\. DaiExploration\-driven optimization for test\-time large language model reasoning\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=NiINDlzvNj)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. WattenbergInference\-time intervention: eliciting truthful answers from a language model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 41451–41530\.External Links:[Document](https://dx.doi.org/10.52202/075280-1797),[Link](https://doi.org/10.52202/075280-1797)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)S\. Li, Z\. Zhou, W\. Lam, C\. Yang, and C\. LuRePO: replay\-enhanced policy optimization\.External Links:2506\.09340,[Document](https://dx.doi.org/10.48550/arXiv.2506.09340),[Link](https://arxiv.org/abs/2506.09340)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p2.1),[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px2.p1.1)\.
- Liaoet al\.\(2025\)M\. Liao, X\. Xi, R\. Chen, J\. Leng, Y\. Hu, K\. Zeng, S\. Liu, and H\. WanEnhancing efficiency and exploration in reinforcement learning for LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 1451–1463\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.75),[Link](https://aclanthology.org/2025.emnlp-main.75/)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p2.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.arXiv preprint arXiv:2305\.20050\.External Links:[Link](https://arxiv.org/abs/2305.20050)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding R1\-Zero\-like training: a critical perspective\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=5PAF7PAY2Y)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px2.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p5.1),[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof Q&A benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Rimskyet al\.\(2024\)N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. TurnerSteering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 15504–15522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.828),[Link](https://doi.org/10.18653/v1/2024.acl-long.828)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300),[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p1.1),[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: a flexible and efficient RLHF framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.External Links:[Document](https://dx.doi.org/10.1145/3689031.3696075),[Link](https://doi.org/10.1145/3689031.3696075)Cited by:[§C\.1](https://arxiv.org/html/2610.27012#A3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px3.p1.1)\.
- Siniiet al\.\(2026\)V\. Sinii, N\. Balagansky, G\. Gerasimov, D\. Laptev, Y\. Aksenov, V\. Kurochkin, A\. Gorbatovski, B\. Shaposhnikov, and D\. GavrilovSmall vectors, big effects: a mechanistic study of RL\-induced reasoning via steering vectors\.External Links:2509\.06608,[Document](https://dx.doi.org/10.48550/arXiv.2509.06608),[Link](https://arxiv.org/abs/2509.06608)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Stolfoet al\.\(2025\)A\. Stolfo, V\. Balachandran, S\. Yousefi, E\. Horvitz, and B\. NushiImproving instruction\-following in language models through activation steering\.InInternational Conference on Learning Representations,pp\. 55790–55823\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/8c3262a4c965ba9888f120d4f9e13478-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.2 edition,MIT Press\.External Links:[Link](http://incompleteideas.net/book/the-book-2nd.html)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p1.1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. WeiChallenging BIG\-Bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824),[Link](https://doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Turneret al\.\(2023\)A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmidSteering language models with activation engineering\.External Links:2308\.10248,[Document](https://dx.doi.org/10.48550/arXiv.2308.10248),[Link](https://arxiv.org/abs/2308.10248)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Venhoffet al\.\(2025\)C\. Venhoff, I\. Arcuschin, P\. Torr, A\. Conmy, and N\. NandaUnderstanding reasoning in thinking language models via steering vectors\.InICLR 2025 Workshop on Reasoning and Planning for Large Language Models,External Links:2506\.18167,[Link](https://iclr.cc/virtual/2025/32763)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang, Y\. Liu, A\. Yang, A\. Zhao, Y\. Yue, S\. Song, B\. Yu, G\. Huang, and J\. LinBeyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for LLM reasoning\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 115452–115486\.External Links:[Document](https://dx.doi.org/10.52202/085713-3850),[Link](https://doi.org/10.52202/085713-3850)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 95266–95290\.External Links:[Document](https://dx.doi.org/10.52202/079017-3018),[Link](https://doi.org/10.52202/079017-3018)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, R\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 113222–113244\.External Links:[Document](https://dx.doi.org/10.52202/085713-3775),[Link](https://doi.org/10.52202/085713-3775)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p1.1),[§1](https://arxiv.org/html/2610.27012#S1.p2.1),[§1](https://arxiv.org/html/2610.27012#S1.p5.1),[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px2.p1.1)\.
- Yueet al\.\(2025\)Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. HuangDoes reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 57654–57689\.External Links:[Document](https://dx.doi.org/10.52202/085713-1933),[Link](https://doi.org/10.52202/085713-1933)Cited by:[§1](https://arxiv.org/html/2610.27012#S1.p1.1),[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1)\.
- Zhang and Math\-AI Team \(2024\)Y\. Zhang and Math\-AI TeamAmerican invitational mathematics examination \(AIME\) 2024\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime24)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Zhang and Math\-AI Team \(2025\)Y\. Zhang and Math\-AI TeamAmerican invitational mathematics examination \(AIME\) 2025\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime25)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2025a\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.External Links:2507\.18071,[Link](https://arxiv.org/abs/2507.18071)Cited by:[§4\.1](https://arxiv.org/html/2610.27012#S4.SS1.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2025b\)H\. Zheng, Y\. Zhou, B\. Bartoldson, B\. Kailkhura, F\. Lai, J\. Zhao, and B\. ChenAct only when it pays: efficient reinforcement learning for LLM reasoning via selective rollouts\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 124321–124346\.External Links:[Document](https://dx.doi.org/10.52202/085713-4143),[Link](https://doi.org/10.52202/085713-4143)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px1.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. HendrycksRepresentation engineering: a top\-down approach to AI transparency\.External Links:2310\.01405,[Document](https://dx.doi.org/10.48550/arXiv.2310.01405),[Link](https://arxiv.org/abs/2310.01405)Cited by:[§2](https://arxiv.org/html/2610.27012#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AComplete Training Algorithm

Algorithm[1](https://arxiv.org/html/2610.27012#algorithm1)summarizes the implementation of ASE\-GRPO\.

Algorithm 1ASE\-GRPO\.Input:initial policyπθ\(0\)\\pi\_\{\\theta^\{\(0\)\}\}, training distribution𝒟\\mathcal\{D\}, rollout countNN, bank sizeKK, steering strengthα\\alpha, EMA coefficientβ\\beta, minimum pair countBminB\_\{\\min\}, and update intervalΔ\\Delta\.

Initialize:𝒱\(0\)←∅\\mathcal\{V\}^\{\(0\)\}\\leftarrow\\emptysetand paired\-activation bufferℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset\.

1. 1\.Fort=0,1,2,…t=0,1,2,\\ldots, sample a batch of problems from𝒟\\mathcal\{D\}and generate routed rollouts withπθ\(t\)\\pi\_\{\\theta^\{\(t\)\}\}and𝒱\(t\)\\mathcal\{V\}^\{\(t\)\}\.
2. 2\.Compute binary verifier rewards, then balance and pair correct and incorrect trajectories within each group\.
3. 3\.Replay the selected trajectories with the unsteered actor, read the corresponding activations, and append the paired activations toℬ\\mathcal\{B\}\.
4. 4\.If\|ℬ\|≥Bmin\|\\mathcal\{B\}\|\\geq B\_\{\\min\}and at leastΔ\\Deltaiterations have elapsed, fit PCA on the pooled activations and run K\-means on the PCA\-space pair differences\. Compute cluster means in the original hidden space, match candidates to historical directions, apply EMA and normalization, stage𝒱\(t\+1\)\\mathcal\{V\}^\{\(t\+1\)\}, and clearℬ\\mathcal\{B\}\.
5. 5\.Compute group\-relative advantages and update the base policy with the clipped GRPO objective\.
6. 6\.Synchronize actor weights to the rollout engine\. Activate a staged vector bank if one exists; otherwise set𝒱\(t\+1\)←𝒱\(t\)\\mathcal\{V\}^\{\(t\+1\)\}\\leftarrow\\mathcal\{V\}^\{\(t\)\}\.

Output:the base policy; discard the vector bank\.

## Appendix BEvaluation and Checkpoint Selection

Training uses only the local DAPO\-Math\-17k training file\. The original JSONL contains 17,398 samples; filtering by maximum prompt length retains 17,390 samples in the standard configuration, while RePO uses 17,344 samples under its fixed rollout configuration\.

We use MATH500 for test\-set adaptive model selection by evaluating it periodically during training\. For ASE and all baselines, we select the checkpoint with the highest MATH500 Mean@8\. We evaluate MATH500 every 100 optimization steps using eight responses per problem, temperature 0\.6, top\-pp0\.95, top\-k=0k=0, and seed 42\. Evaluation on Minerva, AIME24, AIME25, AMC24, OlympiadBench, and the out\-of\-distribution benchmarks \(MMLU\-Pro, GPQA\-Diamond, and Big\-Bench Hard\) is performed only on the selected checkpoint for each experiment\.

## Appendix CImplementation Details

### C\.1Common training and evaluation settings

Table 7:Common settings for the two model sizes\.All training uses verl\([Sheng et al\., 2025](https://arxiv.org/html/2610.27012#bib.bib27)\), vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.27012#bib.bib10)\)rollouts, BF16, and FlashAttention\-2\([Dao, 2024](https://arxiv.org/html/2610.27012#bib.bib40)\)\. We filter samples exceeding the maximum prompt length\. Except for method\-specific objectives or reward processing, we use neither KL loss nor entropy regularization\.

### C\.2Baseline and ASE configurations

Table 8:Method\-specific configurations\.Table 9:ASE\-GRPO configuration\.This subsection supplements the common settings above with method\-specific configurations for the baselines and ASE\-GRPO\. The two tables summarize the policy optimization settings and extra processing used by each method, together with the activation readout, routing, steering, and vector\-bank update settings used by ASE\-GRPO\. For rollout correction, we use verl’s default token\-level truncated importance\-sampling configuration, withcIS=2\.0c\_\{\\mathrm\{IS\}\}=2\.0and no batch normalization of the weights\. For layer choice, we read out activations and apply steering at the output of Transformer block 0 in all ASE experiments\. We use a single intervention layer to avoid accumulating direct vector additions across multiple depths, while an early intervention allows the injected direction to propagate through the remaining blocks\. This choice is heuristic: intermediate or later layers may provide more informative representations for constructing exploration directions\. We did not run a layer sweep within the available compute budget, so we do not claim that block 0 is optimal; comparing readout and intervention layers, including multi\-layer configurations, remains future work\.

## Appendix DAdditional Ablation

### D\.1Hyperparameter Sensitivity

We report ASE sensitivity sweeps for the number of clustersKK, the EMA coefficientβ\\beta, the steering strengthα\\alpha, and the response\-relative activation readout positionpp\. Each run uses its best validation checkpoint\. Entries are Mean@8 / Pass@8 percentages; the mathematical average covers the six mathematical benchmarks and the OOD average covers MMLU\-Pro, GPQA\-D, and BBH\. The sweeps vary one parameter at a time, with the remaining parameters fixed as indicated in each table\.

Table 10:Sensitivity to the number of clustersKK\. For this ASE sweep, we fixα=0\.4\\alpha=0\.4andβ=0\.5\\beta=0\.5\.Table 11:Sensitivity to the EMA coefficientβ\\beta\. For this ASE sweep, we fixK=3K=3andα=0\.2\\alpha=0\.2\.Table 12:Sensitivity to the steering strengthα\\alpha\. For this ASE sweep, we fixK=3K=3andβ=0\.5\\beta=0\.5\.The three sweeps suggest that a small route bank is sufficient under the tested rollout budget\. For the cluster\-count sweep,K=3K=3gives the highest mathematical Mean@8 and Pass@8, at 29\.64 and 47\.02, respectively, whereasK=2K=2gives the highest OOD Mean@8 and Pass@8, at 40\.08 and 78\.44\. IncreasingKKbeyond three does not produce a monotonic improvement: both mathematical metrics are lower forK∈\{4,5,6\}K\\in\\\{4,5,6\\\}than forK=3K=3\. This indicates that adding routes alone does not guarantee better coverage, and that the available rollout evidence must also support reliable estimation of each direction\.

The EMA sweep supports using temporal smoothing when updating the vector bank\. Relative to the unsmoothed settingβ=0\\beta=0, the intermediate settingβ=0\.5\\beta=0\.5improves mathematical Mean@8 from 26\.48 to 30\.21 and OOD Mean@8 from 35\.84 to 40\.79\. It also gives the highest OOD Pass@8, 79\.71, while the highest mathematical Pass@8 is obtained atβ=0\.7\\beta=0\.7, with 48\.83\. Thus, moderate\-to\-strong smoothing performs well across the reported metrics, withβ=0\.5\\beta=0\.5providing a useful overall operating point rather than being the strict optimum for every metric\.

The strength sweep shows non\-monotonic variation across the tested values ofα\\alpha\. Mathematical Mean@8 ranges from 28\.38 to 30\.31 and Pass@8 from 44\.63 to 48\.67; OOD Mean@8 ranges from 38\.75 to 41\.35 and Pass@8 from 76\.52 to 79\.97\. All four metrics reach their highest values atα=0\.5\\alpha=0\.5: 30\.31/48\.67 for Math Avg\. and 41\.35/79\.97 for OOD Avg\. Increasing the strength toα=0\.6\\alpha=0\.6lowers all four metrics, so stronger steering does not consistently improve the final unsteered policy over this range\.

These tables are separate one\-parameter sensitivity analyses rather than a joint factorial study\. In particular, the cluster\-count sweep fixesα=0\.4\\alpha=0\.4, the EMA sweep fixesα=0\.2\\alpha=0\.2, and the strength sweep variesα\\alphawithK=3K=3andβ=0\.5\\beta=0\.5\. The different fixed strengths reflect separate experimental runs that inherited the working configuration used at the time of each sweep, rather than a rerun of every sweep after selecting a single global anchor\. This design keeps each table’s within\-sweep comparison well\-defined, but it prevents comparisons of absolute values across tables from being interpreted as a unified hyperparameter ranking\. In particular, the best row in each table should not be combined to claim a jointly optimal\(K,β,α\)\(K,\\beta,\\alpha\)configuration\. A fully matched sensitivity study would hold one common anchor fixed for all three scans, or would evaluate a prespecified joint grid; the present results are intended to characterize local sensitivity around the configurations used in the respective runs\.

We also vary the response\-relative readout position on Qwen3\-1\.7B\-Base in a separate sweep, selecting activations at a fractionffof each response’s length\. Its absolute scores are not directly comparable with those of the preceding ablations since the sweeps were not conducted under a common matched configuration\. Within this sweep,f=0\.25f=0\.25performs best on the mathematical benchmarks, the primary target of ASE\. One possible explanation is that earlier response states reflect the initial solution strategy before later calculations and answer formatting add trajectory\-specific variation, making success–failure contrasts more reusable across problems\. This explanation remains a hypothesis based on single\-run comparisons\.

Table 13:Sensitivity to the response\-relative activation readout position\. We fixK=3K=3,α=0\.2\\alpha=0\.2, andβ=0\.7\\beta=0\.7; scores are Mean@8 / Pass@8 \(%\)\. Math Avg\. and OOD Avg\. use the same benchmark groups as the main tables\.
### D\.2Ablating Verifier\-Guided Direction

The matched controls in Table[4](https://arxiv.org/html/2610.27012#S4.T4)change direction construction while retaining the clusteredK=3K=3bank, routing, and EMA updates\. For random directions, each independent random vector is rescaled to the corresponding cluster candidate’s originalL2L\_\{2\}norm before the EMA update\. For shuffled labels, the positive and negative labels within each activation pair are randomly exchanged before PCA, K\-means, and cluster\-direction estimation\. Consequently, the controls’ gains over GRPO do not isolate the effect of arbitrary activation perturbations: the shared bank structure and temporal averaging may also contribute\.

To examine this possibility, we additionally use unstructured random steering: eight sampled unit random vectors, one per rollout, with no clustering or EMA\. The vectors do not depend on activations or verifier labels\. As shown in Table[14](https://arxiv.org/html/2610.27012#A4.T14), this control reaches 26\.65/41\.59 on Math Avg\., compared with 26\.00/43\.91 for GRPO and 30\.31/48\.67 for ASE\-GRPO\. Its Mean@8 is only 0\.65 points above GRPO, while its Pass@8 is 2\.32 points lower, so arbitrary random steering does not yield a consistent improvement in this comparison\. This pattern is consistent with the structured bank and EMA contributing to the gains of the matched controls\.

Table 14:Six\-benchmark Math Avg\. \(Mean@8 / Pass@8, %\)\. GRPO and ASE\-GRPO use the first evaluation at their selected checkpoints\.
### D\.3Inference\-time steering sweep

The sweep compares three checkpoints: Qwen3\-1\.7B\-Base and its GRPO\- and ASE\-GRPO\-trained variants\. For each model, vectors are constructed from response\-relative activations captured on the 17,390 filtered DAPO\-Math\-17k prompts\. We use block 0, within\-problem verifier\-reward pairing, CAA\-style positive\-minus\-negative differences, no cluster\-feature scaling, PCA dimension 64, K\-meansn\_init=512, balanced\-uniform route allocation, and a clean route\. We evaluate

K∈\{1,2,3,4,5,6,7\},f∈\{0\.25,0\.50,0\.75\},α∈\{0\.25,0\.50,0\.75,1\.00\}\.K\\in\\\{1,2,3,4,5,6,7\\\},\\qquad f\\in\\\{0\.25,0\.50,0\.75\\\},\\qquad\\alpha\\in\\\{0\.25,0\.50,0\.75,1\.00\\\}\.\(11\)This gives 84 steering conditions per model\. Each steering condition is independently repeated three times, with eight responses per problem\. The Base and GRPO clean references use three standalone evaluations; the ASE\-GRPO clean reference combines its selected\-checkpoint training validation with two standalone evaluations\. Evaluation uses MATH500, Minerva, AIME24, AIME25, and AMC24, temperature 0\.6, top\-pp0\.95, top\-k=0k=0, maximum response length 4096, and thinking mode\.

For a sweep cell, Core Math Mean@8 and Pass@8 are the unweighted averages across the five benchmarks\. We report the mean and sample standard deviation over the three evaluation repeats\. The corresponding clean\-relative changes use the same model’s clean mean:

Δmean=Mean​@​8¯steer−Mean​@​8¯clean,Δpass=Pass​@​8¯steer−Pass​@​8¯clean\.\\Delta\_\{\\mathrm\{mean\}\}=\\overline\{\\mathrm\{Mean@8\}\}\_\{\\mathrm\{steer\}\}\-\\overline\{\\mathrm\{Mean@8\}\}\_\{\\mathrm\{clean\}\},\\qquad\\Delta\_\{\\mathrm\{pass\}\}=\\overline\{\\mathrm\{Pass@8\}\}\_\{\\mathrm\{steer\}\}\-\\overline\{\\mathrm\{Pass@8\}\}\_\{\\mathrm\{clean\}\}\.\(12\)Table[3](https://arxiv.org/html/2610.27012#S4.T3)uses the highest Core Math Mean@8 configuration from each model’s sweep:\(K,f,α\)=\(7,0\.50,0\.50\)\(K,f,\\alpha\)=\(7,0\.50,0\.50\)for Base,\(7,0\.25,0\.25\)\(7,0\.25,0\.25\)for GRPO, and\(4,0\.75,0\.50\)\(4,0\.75,0\.50\)for ASE\-GRPO\. These sweep\-selected results are interpreted as exploratory upper envelopes\.

Table 15:Inference\-time sweep grid and reporting units\.