GAPS: Dimension-Level Gates for Conditional Activation Steering

# GAPS: Dimension-Level Gates for Conditional Activation Steering
Source: [https://arxiv.org/html/2609.01878](https://arxiv.org/html/2609.01878)
###### Abstract

Activation steering suppresses undesired behaviors in language models by adding a steering vector to the hidden state during generation\. Recent conditional methods such as CAST and DSAS improve the behavior–capability trade\-off by deciding when to intervene, but once active, they apply the full dense vector to all hidden dimensions, regardless of whether a neuron carries concept information or already lies in the desired regime\. We introduce dimension\-level conditioning as a complementary axis of selectivity that also decides which neurons to intervene on\. Our method, GAPS \(Gated Activation steering via Posterior and Separability\), combines two training\-free gates: a static separability gate that restricts steering to neurons with statistically reliable concept information \(via AUROC\), and a dynamic posterior gate that steers a neuron only when its current activation is better explained by the undesired concept under a Gaussian model\. The gates add O\(D\) overhead per token, and they plug into existing conditional methods\. On toxicity mitigation \(RealToxicityPrompts\) and concept removal \(OneSeC\) with Gemma\-3 \(4B\) and Qwen\-3 \(1\.7B\), GAPS consistently matches or improves the Pareto front of its token\-level counterparts; under a fixed capability budget, DSAS\+GAPS reduces Gemma\-3’s toxicity rate from 6\.52% to 0\.48%, versus 3\.52% for DSAS alone\. Ablations attribute most of the gain to the posterior gate\.

## 1Introduction

Language models often need to avoid unwanted outputs, such as toxic continuations or references to a specific concept, but retraining the model for every new constraint is impractical\. Activation steering offers a low\-cost alternative: a steering vector, typically computed as the mean difference between hidden states collected from examples of desired and undesired behavior, is added to the model’s hidden state during generation\([Rimsky et al\., 2024](https://arxiv.org/html/2609.01878#bib.bib1);[Suau et al\., 2024](https://arxiv.org/html/2609.01878#bib.bib4);[Rodriguez et al\., 2025a](https://arxiv.org/html/2609.01878#bib.bib9);[Rodriguez et al\., 2025b](https://arxiv.org/html/2609.01878#bib.bib14)\)\. This approach requires no gradient updates and adds little inference cost\. However, conventional activation steering applies the same dense vector at every generation step, regardless of whether the current context warrants intervention\. Interventions can therefore degrade fluency and downstream accuracy\. Steering methods are best compared by the trade\-off they achieve between behavior change and capability loss\.

Recent work improves this trade\-off by deciding*when*to intervene: token\-level conditional methods such as CAST\([Lee et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib2)\)and DSAS\([Ferrando et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib3)\)gate or scale the intervention based on the current context, intending to leave benign generation largely untouched; related approaches use probes or lightweight controllers to trigger, scale, or calibrate the intervention\([Li et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib20);[Wang et al\., 2025a](https://arxiv.org/html/2609.01878#bib.bib21);[Cheng et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib23);[Hegazy et al\., 2026](https://arxiv.org/html/2609.01878#bib.bib19)\)\.

However, whether conditional or not, the intervention itself remains dense in space: whenever steering is applied, the full vector is added to allDDneurons of the hidden state\. We posit that this is more than the task requires, for two reasons\. First, many neurons carry little or no information about the concept, so shifting them cannot suppress the undesired behavior and only perturbs the representation\. Second, a neuron that does encode the concept may already be in the desired regime at the current step; pushing a non\-toxic activation further in the non\-toxic direction does not make the output less toxic, but it does move the representation away from the model’s familiar activation distribution\.

In this work, we add a second axis of conditioning\. Token\-level methods decide when to intervene; we also dynamically decide which neurons in the hidden state to intervene on, with the goal of changing the hidden state no more than the steering objective requires\. We do this with two dimension\-level gates \(Figure[1](https://arxiv.org/html/2609.01878#S2.F1)\)\. The first, a static*separability gate*, restricts steering to only the neurons that carry discernible information about the target concept\. For each neuron, we measure how well its activations separate the two concepts with the Area under the ROC, and keep the neuron only if separable enough\. The second, a dynamic*posterior gate*, decides at each generation step whether a neuron is currently expressing the undesired behavior\. We model the concept\-conditional activation distribution of each neuron as a Gaussian and steer a neuron only if its current activation is more likely under the undesired concept than under the desired one\.

The separability gate is computed once from the contrastive data and fixes the set of neurons that can encode the behavior at all\. The posterior gate is evaluated at every generation step and decides whether those neurons are currently expressing the undesired behavior\. We refer to the combined gates as GAPS \(Gated Activation steering via Posterior and Separability\)\. GAPS strictly generalizes prior methods: with both dimension\-level gates set to one, it reduces to CAST or DSAS, and with the token\-level gate also set to one, it reduces to unconditional steering\. The gates costO⁡\(D\)O\(D\)elementwise operations per token, the same order as the steering vector addition itself, so dimension\-level conditioning adds only a constant\-factor overhead to canonical steering, and all required statistics are estimated from the same contrastive activations used to construct the steering vector\.

We evaluate on toxicity mitigation with RealToxicityPrompts\([Gehman et al\., 2020](https://arxiv.org/html/2609.01878#bib.bib13)\)and concept removal over seven OneSeC concepts\([Scarlini et al\., 2019](https://arxiv.org/html/2609.01878#bib.bib15)\), using Gemma\-3 \(4B\) and Qwen\-3 \(1\.7B\) with CAST and DSAS as token\-level baselines, sweeping the intervention strength to trace the trade\-off against Wikipedia perplexity and MMLU\([Wikimedia Foundation,](https://arxiv.org/html/2609.01878#bib.bib17);[Hendrycks et al\., 2021](https://arxiv.org/html/2609.01878#bib.bib12)\)\. Across all eight combinations of model, token\-level method, and task, GAPS matches or improves the Pareto front of its token\-level counterpart\. Under a fixed capability budget, DSAS\+GAPS lowers Gemma\-3’s toxicity rate from 6\.52% to 0\.48%, versus 3\.52% for DSAS alone\. Ablations attribute most of the gain to the posterior gate, with the separability gate a cheap complement: the strongest operating point requires both\.

Our contributions are as follows:

- •We introduce dimension\-level conditioning as a second axis of selectivity in activation steering: beyond deciding*when*to steer, we*dynamically*decide*which*neurons to steer\.
- •We propose GAPS, a pair of training\-free gates: a static separability gate that restricts steering to neurons with reliable concept information, and a dynamic posterior gate that steers a neuron only when its activation is better explained by the undesired concept\. GAPS plugs into existing conditional methods and strictly generalizes them\.
- •Across two models, two token\-level methods, and two tasks, GAPS consistently matches or improves the Pareto front of its token\-level counterparts; ablations attribute most of the gain to the posterior gate\.

## 2Preliminaries

Figure 1:The two dimension\-level gates of GAPS\.The static separability gate𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}identifies which neurons encode the concept, keeping only neurons that separate the target and source concepts \(computed offline\)\. The dynamic posterior gate𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}detects whether the source \(undesired\) concept is*currently*expressed, steering a neuron only when its current activation is more likely under the source than the target concept\. Their product gives the final per\-neuron mask applied to the steering vector\.Neuron\.A neuron refers to a component of a hidden state vector in a transformer layer\. We writeh∈ℝDh\\in\\mathbb\{R\}^\{D\}for the hidden state, where neuronddcorresponds to thedd\-th coordinate, ford∈\{1,…,D\}d\\in\\\{1,\\dots,D\\\}\. During generation,hth\_\{t\}denotes the hidden state at generation steptt, andht,dh\_\{t,d\}denotes the activation value of neuronddat that step\.

Activation steering\.Given a steering objective \(e\.g\., non\-toxic generation\), we collect hidden\-state activations from target examples \(tar\\mathrm\{tar\}\), which represent the behavior we steer toward, and source examples \(src\\mathrm\{src\}\), which represent the behavior we steer away from\. During generation, standard activation steering adds a fixed direction to the hidden state:ht←ht\+α​vh\_\{t\}\\leftarrow h\_\{t\}\+\\alpha\\,v, whereα\>0\\alpha\>0controls the intervention strength\. We use the standard mean\-difference direction\([Rimsky et al\., 2024](https://arxiv.org/html/2609.01878#bib.bib1)\):

v=μtar−μsrc,μc=1Nc​∑i=1Nchic,c∈\{tar,src\},v=\\mu^\{\\mathrm\{tar\}\}\-\\mu^\{\\mathrm\{src\}\},\\;\\mu^\{c\}=\\frac\{1\}\{N\_\{c\}\}\\sum\_\{i=1\}^\{N\_\{c\}\}h^\{c\}\_\{i\},\\;c\\in\\\{\\mathrm\{tar\},\\mathrm\{src\}\\\},wherehich^\{c\}\_\{i\}is the activation extracted from theii\-th example of conceptcc\.

Token\-level conditional steering\.Rather than applying the intervention at every generation step \(every token\), token\-level conditional methods modulate its strength using a scalar gateg⁡\(ht\)∈\[0,1\]g\(h\_\{t\}\)\\in\[0,1\]:

ht←ht\+α​g​\(ht\)​vh\_\{t\}\\leftarrow h\_\{t\}\+\\alpha\\,g\(h\_\{t\}\)\\,v\(1\)We consider two such methods\.CAST\([Lee et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib2)\)uses a hard binary gate obtained by thresholding the alignment between the current hidden state and an extracted condition direction\.DSAS\([Ferrando et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib3)\), on the other hand, uses the output of a probe as a continuous gate that scales the intervention according to the detected strength of the undesired behavior\.

## 3Neuron\-Level Conditional Steering

Token\-level conditional methods \(Eq\.[1](https://arxiv.org/html/2609.01878#S2.E1)\) address a key weakness of unconditional steering: by modulating the intervention with a token\-level gateg⁡\(ht\)g\(h\_\{t\}\), they intervene only when the running context actually exhibits the source \(undesired\) behavior, leaving benign generation untouched and thereby reducing unnecessary capability loss\. BothCASTandDSASthus determine*when*to steer\. However, once the gate is active, they apply the full dense steering vectorvvto allDDneurons, regardless of whether a neuron carries concept\-discriminative information at all, or whether its current activation is already consistent with the target \(desired\) behavior\. Steering such neurons cannot reduce the source behavior; it can only cause unnecessary perturbation of the model\. Our method adds neuron\-level selectivity to address these two limitations: a static separability gate determines which neurons can encode the behavior, and a dynamic posterior gate determines whether intervention on these neurons is warranted*now*\(Figure[1](https://arxiv.org/html/2609.01878#S2.F1)\)\.

Static separability gate\.For each neurond∈\{1,…,D\}d\\in\\\{1,\\dots,D\\\}, we measure how well its activation distribution separates the two concepts using the area under the ROC curve,AUROCd=Pr⁡\(hdsrc\>hdtar\)\\mathrm\{AUROC\}\_\{d\}=\\Pr\(h^\{\\mathrm\{src\}\}\_\{d\}\>h^\{\\mathrm\{tar\}\}\_\{d\}\), estimated from the contrastive sets; a value of12\\tfrac\{1\}\{2\}indicates no concept information\. Computing it via the Mann–WhitneyUUidentity\([Mann and Whitney, 1947](https://arxiv.org/html/2609.01878#bib.bib5);[Hanley and McNeil, 1982](https://arxiv.org/html/2609.01878#bib.bib6);[Mason and Graham, 2002](https://arxiv.org/html/2609.01878#bib.bib7)\)also yields the exact mean and standard deviation of the estimate under the null hypothesis of no concept signal:AUROCd\\mathrm\{AUROC\}\_\{d\}has mean12\\tfrac\{1\}\{2\}and standard deviation

σ0=Nsrc\+Ntar\+112​Nsrc​Ntar,\\sigma\_\{0\}\\;=\\;\\sqrt\{\\frac\{N\_\{\\mathrm\{src\}\}\+N\_\{\\mathrm\{tar\}\}\+1\}\{12\\,N\_\{\\mathrm\{src\}\}N\_\{\\mathrm\{tar\}\}\}\},\(2\)
whereNcN\_\{c\}denotes the number of contrastive examples of conceptc∈\{src,tar\}c\\in\\\{\\mathrm\{src\},\\mathrm\{tar\}\\\}\.

We retain neuronddonly if its deviation from chance exceeds a two\-sided threshold ofτz\\tau\_\{z\}null standard deviations,

𝐦dsep=\[\|AUROCd−12\|\>τzσ0\],\\mathbf\{m\}^\{\\mathrm\{sep\}\}\_\{d\}\\;=\\;\\mathbf\{1\}\\\!\\left\[\\bigl\|\\mathrm\{AUROC\}\_\{d\}\-\\tfrac\{1\}\{2\}\\bigr\|\>\\tau\_\{z\}\\sigma\_\{0\}\\right\],\(3\)
whereτz\\tau\_\{z\}controls the stringency of the gate\. Unlike a fixed AUROC cutoff, whose statistical meaning depends on sample size, this criterion adapts to sample size: with fewer examples, a neuron must have an AUROC farther from0\.50\.5to be retained, whereas with more examples, smaller but statistically reliable deviations suffice\. We setτz=7\\tau\_\{z\}=7, well above thez≈5\.1z\\approx 5\.1required for Bonferroni\-corrected significance atα0=10−3\\alpha\_\{0\}=10^\{\-3\}across allDDneurons, so the gate retains only neurons whose concept signal is statistically unambiguous \(more details in Appendix[G](https://arxiv.org/html/2609.01878#A7)\)\. WithNsrc=Ntar=700N\_\{\\mathrm\{src\}\}=N\_\{\\mathrm\{tar\}\}=700\(matching our OneSeC setup\), this corresponds to retaining only neurons whose AUROC lies outside approximately\[0\.39,0\.61\]\[0\.39,0\.61\]\. The test is two\-sided because a neuron may encode the source concept through either larger or smaller activations; the appropriate steering direction is captured byvv\.

Dynamic posterior gate\.The static gate identifies which neurons ever encode the concept; it does not tell us whether a given neuron requires intervention*now*\. LetC=\{ct​a​r,cs​r​c\}C=\\\{c^\{tar\},c^\{src\}\\\}denote the target and source concepts\. Our goal is to steer neuronddonly when its current activation valueht,dh\_\{t,d\}is more likely under the source conceptcsrcc^\{\\mathrm\{src\}\}than under the target conceptctarc^\{\\mathrm\{tar\}\}\. Using Bayes’ theorem, we compute the posterior probability that a given activationht,dh\_\{t,d\}arose fromcs​r​cc^\{src\}, under the assumption that both concepts are a priori equally likely:

p⁡\(csrc∣ht,d\)=p⁡\(ht,d∣csrc\)​p​\(csrc\)∑c∈Cp⁡\(ht,d∣c\)​p​\(c\)=equal priors​p⁡\(ht,d∣csrc\)p⁡\(ht,d∣csrc\)\+p⁡\(ht,d∣ctar\)\.\\begin\{aligned\} p\\bigl\(c^\{\\mathrm\{src\}\}\\mid h\_\{t,d\}\\bigr\)&=\\frac\{p\\bigl\(h\_\{t,d\}\\mid c^\{\\mathrm\{src\}\}\\bigr\)p\\bigl\(c^\{\\mathrm\{src\}\}\\bigr\)\}\{\\displaystyle\\sum\_\{c\\in C\}p\\bigl\(h\_\{t,d\}\\mid c\\bigr\)p\(c\)\}\\\\\[3\.0pt\] &\\overset\{\\text\{equal priors\}\}\{=\}\\frac\{p\\bigl\(h\_\{t,d\}\\mid c^\{\\mathrm\{src\}\}\\bigr\)\}\{p\\bigl\(h\_\{t,d\}\\mid c^\{\\mathrm\{src\}\}\\bigr\)\+p\\bigl\(h\_\{t,d\}\\mid c^\{\\mathrm\{tar\}\}\\bigr\)\}\.\\end\{aligned\}

\(4\)
Directly estimating the two concept\-conditional activation densities,p⁡\(ht,d∣csrc\)p\(h\_\{t,d\}\\mid c^\{\\mathrm\{src\}\}\)andp⁡\(ht,d∣ctar\)p\(h\_\{t,d\}\\mid c^\{\\mathrm\{tar\}\}\), using non\-parametric methods \(e\.g\., kernel density or histogram estimators\) is unreliable given the limited number of contrastive activation samples per neuron and would incur additional per\-token cost during generation\. Therefore, following the finding in\([Haider et al\., 2026](https://arxiv.org/html/2609.01878#bib.bib8);[Fereidouni et al\., 2026](https://arxiv.org/html/2609.01878#bib.bib16)\)that concept\-related neuron activations follow approximately Gaussian distributions, we adopt a parametric Gaussian model for each concept\-conditional density:

p⁡\(ht,d∣c\)=𝒩⁡\(ht,d,μdc,\(σdc\)2\),c∈C,p\\bigl\(h\_\{t,d\}\\mid c\\bigr\)\\;=\\;\\mathcal\{N\}\\\!\\bigl\(h\_\{t,d\};\\mu\_\{d\}^\{c\},\(\\sigma\_\{d\}^\{c\}\)^\{2\}\\bigr\),\\quad c\\in C,\(5\)
whereμdc=\[μc\]d\\mu\_\{d\}^\{c\}=\[\\mu^\{c\}\]\_\{d\}is the mean activation of neuronddfor conceptcc, and\(σdc\)2\(\\sigma\_\{d\}^\{c\}\)^\{2\}is the corresponding variance, both estimated from the contrastive samples belonging to conceptcc\.

The posterior gate then steers a neuron only when its current activation is better explained by the source concept:

𝐦dpost\(ht\)=\[p\(csrc∣ht,d\)\>12\]\.\\mathbf\{m\}^\{\\mathrm\{post\}\}\_\{d\}\(h\_\{t\}\)=\\mathbf\{1\}\\\!\\left\[p\\bigl\(c^\{\\mathrm\{src\}\}\\mid h\_\{t,d\}\\bigr\)\>\\tfrac\{1\}\{2\}\\right\]\.\(6\)
Intuitively, if the current activationht,dh\_\{t,d\}is more likely under the target\-concept distribution than under the source\-concept distribution, pushing it further alongvdv\_\{d\}is unlikely to further suppress the source behavior and may instead introduce an unnecessary perturbation\.

Figure 2:Toxicity mitigation trade\-offs: Wikipedia perplexity vs\. toxicity score \(lower is better on both axes\) for Gemma\-3 4B \(top\) and Qwen\-3 1\.7B \(bottom\), swept over steering strengthα\\alpha\. Each panel shows unconditional steering \(dashed\), a token\-level baseline \(CAST, left; DSAS, right\), and the baseline with our gates \(\+GAPS\); the diamond marks the unsteered model\. \+GAPS consistently shifts the Pareto front toward the bottom\-left\.Figure 3:Concept removal trade\-offs: Wikipedia perplexity vs\. average normalized concept score \(per\-concept score divided by its unsteered value, averaged over seven OneSeC concepts; lower is better on both axes\), swept overα\\alpha\. Panels as in Figure[2](https://arxiv.org/html/2609.01878#S3.F2); shaded bands =±\\pm1 standard error of the mean across concepts\. \+GAPS again matches or improves the trade\-off on both models\.Full update\.Combining the token\-level gate with both neuron\-level conditions yields intervention:

ht←ht\+α​g​\(ht\)​\(𝐦sep⊙𝐦post​\(ht\)\)⊙v\\boxed\{\\;h\_\{t\}\\;\\leftarrow\\;h\_\{t\}\\;\+\\;\\alpha\\;g\(h\_\{t\}\)\\;\\big\(\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\odot\\mathbf\{m\}^\{\\mathrm\{post\}\}\(h\_\{t\}\)\\big\)\\odot v\\;\}

\(7\)where⊙\\odotis the elementwise product,g∈\{gcast,gdsas\}g\\in\\\{g\_\{\\textsc\{cast\}\},g\_\{\\textsc\{dsas\}\}\\\}\. We call the combined gate \(𝐦sep⊙𝐦post\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\\!\\odot\\\!\\mathbf\{m\}^\{\\mathrm\{post\}\}\), GAPS \(yielding CAST\+GAPS and DSAS\+GAPS\)\. During autoregressive generation, the update is applied to the final token position of the running context\. Setting𝐦sep=𝐦post=1\\mathbf\{m\}^\{\\mathrm\{sep\}\}=\\mathbf\{m\}^\{\\mathrm\{post\}\}=1recovers token\-level conditional steering exactly \(CAST or DSAS\), and additionally settingg≡1g\\equiv 1recovers unconditional steering, so Eq\.[7](https://arxiv.org/html/2609.01878#S3.E7)strictly generalizes the prior methods; the sweep overα\\alphatraces the behavior\-vs\.\-capability Pareto curve for every variant\. Offline, all required statistics are estimated in a single pass over the same contrastive activations used to constructvv; online, evaluating both gates addsO⁡\(D\)O\(D\)elementwise operations per token, the same order as the steering\-vector additionα​v\\alpha vitself \(see Appendix[A](https://arxiv.org/html/2609.01878#A1)for details\)\.

It is important to note that the two dimension\-level gates are complementary rather than redundant:𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}is a global statement about which neurons encode the behavior at all, while𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}is a local statement about whether intervention is currently warranted on those neurons\.

## 4Experiments and Results

Table 1:Ablation of the separability gate \(𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}\), posterior gate \(𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}\), and a simpler range gate \(𝐦rng\\mathbf\{m\}^\{\\text\{rng\}\}\) on toxicity mitigation \(left\) and OneSeC concept removal \(right, averaged over 7 concepts\)\. Per method, we report the lowest behavior score over theα\\alphasweep whose capability cost stays within a fixed budget \(perplexity≤\+5%\\leq\+5\\%, MMLU≥−3%\\geq\-3\\%vs\. unsteered\)\. Best score inbold; second\-bestunderlined\.### 4\.1Experimental Setup

Tasks and models\.We evaluate on toxicity mitigation \(RealToxicityPrompts;[Gehman et al\., 2020](https://arxiv.org/html/2609.01878#bib.bib13)\) and concept removal \(seven OneSeC concepts;[Scarlini et al\., 2019](https://arxiv.org/html/2609.01878#bib.bib15)\), using Gemma\-3 4B[Team et al\. \(2025\)](https://arxiv.org/html/2609.01878#bib.bib10)and Qwen\-3 1\.7B[Yang et al\. \(2025\)](https://arxiv.org/html/2609.01878#bib.bib11)\. All interventions are applied at layer 15 of both models; Appendix[E](https://arxiv.org/html/2609.01878#A5)reports results at layer 18\. Full setup details are in Appendix[F](https://arxiv.org/html/2609.01878#A6)\.

Capability metrics\.To assess whether our method preserves the general capabilities of the language model while avoiding unnecessary interventions, we evaluate model performance along two axes: \(1\) perplexity on Wikipedia sentences[Wikimedia Foundation \(\)](https://arxiv.org/html/2609.01878#bib.bib17), which measures fluency and language modeling quality, and \(2\) performance on the Massive Multitask Language Understanding \(MMLU\) benchmark[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.01878#bib.bib12), which measures knowledge and reasoning ability\. Rather than discrete MMLU accuracy, we report the mean probability assigned to the correct option, which provides a smoother and more sensitive measure of capability degradation under intervention\. Both metrics are computed for the unsteered model and after applying each steering method across a range of intervention strengthsα\\alpha, allowing us to trace the full behavior vs capability trade\-off\.

Baselines\.As an additional baseline for dimension\-level conditioning, we compare against a range gatemrngm^\{\\mathrm\{rng\}\}[Haider et al\. \(2026\)](https://arxiv.org/html/2609.01878#bib.bib8), which steers neuronddonly when its current activationht,dh\_\{t,d\}lies within the typical activation range of the source concept:

𝐦drng\(ht\)=\[\|ht,d−μdsrc\|≤2\.5σdsrc\]\.\\mathbf\{m\}^\{\\mathrm\{rng\}\}\_\{d\}\(h\_\{t\}\)=\\mathbf\{1\}\\\!\\left\[\\left\|h\_\{t,d\}\-\\mu\_\{d\}^\{\\mathrm\{src\}\}\\right\|\\leq 2\.5\\,\\sigma\_\{d\}^\{\\mathrm\{src\}\}\\right\]\.\(8\)

### 4\.2Toxicity Mitigation

Large language models are prone to generating toxic continuations when conditioned on adversarial or toxicity\-eliciting prompts[Gehman et al\. \(2020\)](https://arxiv.org/html/2609.01878#bib.bib13)\. Our goal in this setting is selective intervention: the model should be steered away from toxic generations only when the running context actually carries toxic content, while generation on benign contexts should remain untouched\.

We elicit toxic continuations using prompts from the RealToxicityPrompts \(RTP\) dataset[Gehman et al\. \(2020\)](https://arxiv.org/html/2609.01878#bib.bib13)\. We samplek=10k=10continuations per prompt and classify each as toxic or non\-toxic using a RoBERTa\-based toxicity classifier\. This protocol follows common practice in the literature\([Rodriguez et al\., 2025a](https://arxiv.org/html/2609.01878#bib.bib9);[Ferrando et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib3);[Suau et al\., 2024](https://arxiv.org/html/2609.01878#bib.bib4);[Rodriguez et al\., 2025b](https://arxiv.org/html/2609.01878#bib.bib14)\)\. We report the toxicity rate: the fraction of continuations labeled toxic, averaged over prompts and expressed as a percentage\. The protocol is applied to the unsteered model and to every steering configuration, ensuring that toxicity reductions are directly comparable across all setups\.

As shown in Figure[2](https://arxiv.org/html/2609.01878#S3.F2), both token\-level conditional steering baselines improve substantially over unconditional steering\. On both models, DSAS attains lower perplexity than unconditional steering at matched toxicity levels, and CAST exhibits the same pattern, tracing a better Pareto front\. This confirms that deciding*when*to steer already reduces unnecessary capability loss\. Our approach adds a complementary axis of selectivity: it conditions the intervention at the dimension level via the separability gate and posterior gate \(𝐦sep⊙𝐦post\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\odot\\mathbf\{m\}^\{\\mathrm\{post\}\}\), deciding which neurons to steer\. Applying our gates on top of either baseline \(CAST\+GAPS, DSAS\+GAPS\) consistently shifts the Pareto front further toward the bottom\-left: at any given toxicity score, the gated variants result in equal or lower perplexity than their token\-level counterparts, across both Gemma\-3 \(4B\) and Qwen\-3 \(1\.7B\) and both token\-level methods\. In addition to perplexity, we report MMLU against the toxicity score in Figure[4](https://arxiv.org/html/2609.01878#A2.F4)\(Appendix[C](https://arxiv.org/html/2609.01878#A3)\)\. The benefit of our gates is most pronounced for Gemma\-3 \(4B\), where the gated variants retain substantially higher MMLU than token\-level and unconditional steering at matched toxicity levels, degrading more gracefully as the intervention strength increases; on Qwen\-3 \(1\.7B\), all methods perform comparably on MMLU across most of the toxicity range\.

### 4\.3Concept Removal

Beyond toxicity mitigation, we evaluate our method on concept removal\. As in the toxicity setting, the goal is selective intervention: steer away from a source concept \(e\.g\.,*baby*\) only when the running context evokes it, leaving unrelated contexts unaffected\. For the contrastive data, we follow[Rodriguez et al\. \(2025a\)](https://arxiv.org/html/2609.01878#bib.bib9)and use the OneSeC dataset[Scarlini et al\. \(2019\)](https://arxiv.org/html/2609.01878#bib.bib15)with the same seven concepts \(*football*,*cloud*,*baby*,*church*,*book*,*flower*,*balloon*\) and sampling protocol: for each concept, 700 samples of the source concept \(to steer away from\) and 700 samples drawn from the remaining concepts \(to steer toward\)\. To measure removal, we use GPT\-4o\-mini as an LLM judge to determine whether the source concept is present in each generated continuation; the prompt is in Appendix[B](https://arxiv.org/html/2609.01878#A2)\.

Figure[3](https://arxiv.org/html/2609.01878#S3.F3)reports the perplexity–removal trade\-off\. As in the toxicity setting, applying dimension\-level gates on top of either token\-level baseline matches or improves the Pareto front on both models, with the largest gains on Gemma\-3 \(4B\)\.

MMLU results are reported in Figure[5](https://arxiv.org/html/2609.01878#A3.F5)\(Appendix[D](https://arxiv.org/html/2609.01878#A4)\)\. The two capability metrics paint complementary pictures\. On Gemma\-3 \(4B\), where our gates yield the largest perplexity gains, the dimension\-gated variants perform nearly identically to their token\-level counterparts on MMLU\. On Qwen\-3 \(1\.7B\), the pattern reverses: although the perplexity gains were modest, the gated variants retain higher MMLU at matched removal levels for most intervention strengths\. Overall, dimension\-level gating improves the trade\-off along at least one capability axis on both models; more discussion in Appendix[D](https://arxiv.org/html/2609.01878#A4)\.

### 4\.4Analysis of Dimension\-Level Gates

Table[1](https://arxiv.org/html/2609.01878#S4.T1)isolates the contribution of each dimension\-level condition\. For every method, we sweep the steering strengthα\\alphaand report the operating point with the lowest source \(undesired\) concept score whose capability cost stays inside a fixed budget: perplexity at most 5% above, and MMLU correct\-option probability at most 3% below, the unsteered model\. All rows are therefore compared at matched capability cost\. For concept removal, the capability budget is defined once from the unsteered model, and for each of the seven OneSeC concepts, we separately select the steering strength with the lowest concept score within the budget\. We normalize the resulting concept score by its unsteered value, so that the unsteered baseline is 100%, and then average the normalized scores across all seven concepts\. MMLU and perplexity are likewise averaged over the per\-concept operating points\.

This gives88independent settings:22models×\\times22token\-level methods \(CAST, DSAS\)×\\times22tasks\. Throughout,𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}is*static*, computed once from the contrastive sets, whereas𝐦rng\\mathbf\{m\}^\{\\mathrm\{rng\}\}and𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}are both*dynamic*: each is evaluated at the current activationht,dh\_\{t,d\}at every generation step, and the two differ only in their decision rule, membership in the undesired concept’s empirical range \(Eq\.[8](https://arxiv.org/html/2609.01878#S4.E8)\) versus the posteriorp⁡\(csrc∣ht,d\)\>12p\\bigl\(c^\{\\mathrm\{src\}\}\\mid h\_\{t,d\}\\bigr\)\>\\tfrac\{1\}\{2\}\(Eq\.[6](https://arxiv.org/html/2609.01878#S3.E6)\)\. We summarize four findings\.

Overall ranking: posterior\-gated variants perform best\.In all88settings, the best\-performing variant contains the posterior gate, and in77of88, the two posterior\-gated variants occupy*both*the best and second\-best positions \(1515of the1616top\-two slots\)\. The single exception is Gemma\-3 with CAST on toxicity, where𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}alone \(2\.782\.78\) edges out𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}\(3\.063\.06\);𝐦sep⊙𝐦post\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\\!\\odot\\mathbf\{m\}^\{\\mathrm\{post\}\}is still the best variant there \(2\.142\.14\)\. The strongest overall result is obtained by DSAS\+\+GAPS on Gemma\-3, which lowers toxicity from6\.52%6\.52\\%\(unsteered\) to0\.48%0\.48\\%\. By comparison, DSAS alone reduces toxicity only to3\.52%3\.52\\%\. Thus, GAPS achieves a92\.6%92\.6\\%reduction relative to the unsteered model, versus46\.0%46\.0\\%for the token\-level gate alone\.

Posterior vs\. separability: knowing where the concept lives is not the same as knowing where to act\.The posterior gate \(𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}\) outperforms the static separability gate \(𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}\) in77of the88settings, with improvements of up to a41%41\\%relative reduction in score \(toxicity3\.18%→1\.86%3\.18\\%\\\!\\rightarrow\\\!1\.86\\%for Qwen\-3 with DSAS\)\. The two methods answer different questions\. The static gate \(𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}\) identifies dimensions that*can*encode the undesired concept and steers all of them whenever the token\-level gate is active\. The posterior gate \(𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}\) instead identifies dimensions that are*currently*expressing the undesired concept by comparing how well the present activation is explained by the undesired and desired concept distributions\. Its consistent advantage shows that knowing which neurons can encode a behavior is insufficient; effective steering also requires knowing when that encoding is actually active\.

Posterior vs\. range: why range falls short\.The posterior gate consistently outperforms the range gate\. Across all 8 settings,𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}achieves a lower score than𝐦rng\\mathbf\{m\}^\{\\mathrm\{rng\}\}, and combining it with the separability gate \(𝐦sep⊙𝐦post\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\\!\\odot\\mathbf\{m\}^\{\\mathrm\{post\}\}, i\.e\. GAPS\) likewise outperforms𝐦sep⊙𝐦rng\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\\!\\odot\\\!\\mathbf\{m\}^\{\\mathrm\{rng\}\}in every case\. The difference lies in the decision rule\. The range gate \(𝐦rng\\mathbf\{m\}^\{\\mathrm\{rng\}\}\) considers only the undesired\-concept distribution, activating wheneverht,dh\_\{t,d\}falls within2\.52\.5standard deviations of the undesired\-concept mean \(Eq\.[8](https://arxiv.org/html/2609.01878#S4.E8)\)\. As a result, it also steers activations in regions where the desired and undesired distributions overlap, even when the current activation is equally or better explained by the desired concept\. The posterior gate instead compares the two concept\-conditional densities directly and intervenes only when the undesired concept is the more likely explanation, avoiding these unnecessary updates\.

The separability gate is a consistently useful, essentially free complement\.The static separability gate improves whichever decision rule it is paired with\. Applied directly on top of the token\-level baselines,𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}lowers the score in66of88settings; combined with the range gate,𝐦sep⊙𝐦rng\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\\!\\odot\\\!\\mathbf\{m\}^\{\\mathrm\{rng\}\}improves over𝐦rng\\mathbf\{m\}^\{\\mathrm\{rng\}\}in55of88; and combined with the posterior gate,𝐦sep⊙𝐦post\\mathbf\{m\}^\{\\mathrm\{sep\}\}\\\!\\odot\\\!\\mathbf\{m\}^\{\\mathrm\{post\}\}\(GAPS\) improves over𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}in66of88\. Notably, the strongest operating point in Table[1](https://arxiv.org/html/2609.01878#S4.T1)\(toxicity reduced from6\.52%6\.52\\%to0\.48%0\.48\\%with DSAS on Gemma\-3\) is reached only with the separability gate:𝐦post\\mathbf\{m\}^\{\\mathrm\{post\}\}alone plateaus at1\.26%1\.26\\%\. Since𝐦sep\\mathbf\{m\}^\{\\mathrm\{sep\}\}is computed once from the contrastive sets and adds no per\-token cost, it offers these gains at effectively zero overhead\.

## 5Related Works

Activation steering\.Activation steering adds a concept direction to the hidden state during inference, with no gradient updates and little inference cost\. The direction is typically extracted from contrastive examples or probes\([Turner et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib25);[Rimsky et al\., 2024](https://arxiv.org/html/2609.01878#bib.bib1);[Zou et al\., 2023](https://arxiv.org/html/2609.01878#bib.bib24);[Li et al\., 2023](https://arxiv.org/html/2609.01878#bib.bib26)\), while later work optimizes the intervention itself\([Rodriguez et al\., 2025a](https://arxiv.org/html/2609.01878#bib.bib9);[Rodriguez et al\., 2025b](https://arxiv.org/html/2609.01878#bib.bib14)\)\. All of these methods are unconditional and dense: the intervention is applied at every generation step and to all dimensions of the hidden state\.

When to steer\.A first axis of selectivity decides*when*to intervene, per input or per generation step\. CAST\([Lee et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib2)\)applies a hard gate based on the alignment between the current hidden state and an extracted condition direction, and DSAS\([Ferrando et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib3)\)scales the intervention with the output of a trained probe\. Several works follow the same pattern, using probes or controllers to trigger, scale, or calibrate the intervention\([Li et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib20);[Wang et al\., 2025a](https://arxiv.org/html/2609.01878#bib.bib21);[Cheng et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib23);[Hegazy et al\., 2026](https://arxiv.org/html/2609.01878#bib.bib19);[Hedström et al\., 2025](https://arxiv.org/html/2609.01878#bib.bib18)\)\. In all cases, however, the intervention remains dense: the steering vector is applied in full at the chosen site, with no dimension\-level selectivity\.

Static dimension selection\.Some methods restrict the intervention to a subset of dimensions, but this selection is made offline: AurA\([Suau et al\., 2024](https://arxiv.org/html/2609.01878#bib.bib4)\)dampens experts chosen by AUROC, and SADI\([Wang et al\., 2025b](https://arxiv.org/html/2609.01878#bib.bib22)\)applies a top\-KKcontrastive mask\. Neither decides online, per generation step and per neuron, whether steering is currently warranted, the axis of selectivity GAPS introduces\.

## 6Conclusion

We introduced dimension\-level conditioning as a second axis of selectivity in activation steering: beyond deciding when to intervene, our gates decide, at each generation step, which neurons in the hidden state to steer\. The static separability gate restricts steering to neurons carrying reliable concept information, and the dynamic posterior gate intervenes only when a neuron’s current activation is better explained by the undesired concept\. The combined gates, GAPS, are training\-free, add negligible overhead, and strictly generalize existing conditional methods\. Across two models, two token\-level methods, and two tasks, GAPS matches or improves the Pareto front, reducing Gemma\-3’s toxicity from 6\.52% to 0\.48% at matched capability cost\. Ablations attribute most of the gain to the posterior gate: knowing which neurons can encode a concept is not enough; effective steering requires knowing when it is active\.

## Limitations

The dynamic posterior gate assumes equal priors over the source and target concepts \(Eq\.[4](https://arxiv.org/html/2609.01878#S3.E4)\)\. In deployment, a practitioner who expects mostly benign contexts could setp⁡\(csrc\)<12p\(c^\{\\mathrm\{src\}\}\)<\\tfrac\{1\}\{2\}to make the gate more conservative, whereas adversarial settings may warrant the opposite\. We leave a systematic study of prior selection to future work\.

## References

- Chenget al\.\(2025\)Z\. Cheng, J\. Gan, Z\. Jiang, C\. Wang, Y\. Yin, X\. Luo, Y\. Fu, and Q\. GuSteering when necessary: flexible steering large language models with backtracking\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=l75RyRcevf)Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p2.1)\.
- Fereidouniet al\.\(2026\)M\. Fereidouni, M\. U\. Haider, P\. Ju, and A\.B\. SiddiqueEvaluating sparse autoencoders for monosemantic representation\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 5969–5984\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.313/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.313),ISBN 979\-8\-89176\-386\-9Cited by:[§3](https://arxiv.org/html/2609.01878#S3.p8.1)\.
- Ferrandoet al\.\(2025\)A\. Ferrando, X\. Suau, J\. Gonzàlez, and P\. RodriguezDynamically scaled activation steering\.arXiv preprint arXiv:2512\.03661\.Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p2.1),[§2](https://arxiv.org/html/2609.01878#S2.p3.2),[§4\.2](https://arxiv.org/html/2609.01878#S4.SS2.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p2.1)\.
- Gehmanet al\.\(2020\)S\. Gehman, S\. Gururangan, M\. Sap, Y\. Choi, and N\. A\. SmithRealToxicityPrompts: evaluating neural toxic degeneration in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 3356–3369\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.301/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.301)Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p1.1),[§1](https://arxiv.org/html/2609.01878#S1.p6.1),[§4\.1](https://arxiv.org/html/2609.01878#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.01878#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2609.01878#S4.SS2.p2.1)\.
- Haideret al\.\(2026\)M\. U\. Haider, H\. Rizwan, H\. Sajjad, P\. Ju, and A\.B\. SiddiqueNeurons speak in ranges: breaking free from discrete neuronal attribution\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=AukyIhfBuW)Cited by:[§3](https://arxiv.org/html/2609.01878#S3.p8.1),[§4\.1](https://arxiv.org/html/2609.01878#S4.SS1.p3.1)\.
- Hanley and McNeil \(1982\)J\. A\. Hanley and B\. J\. McNeilThe meaning and use of the area under a receiver operating characteristic \(ROC\) curve\.Radiology143\(1\),pp\. 29–36\.External Links:[Document](https://dx.doi.org/10.1148/radiology.143.1.7063747)Cited by:[§3](https://arxiv.org/html/2609.01878#S3.p2.1)\.
- Harriset al\.\(2020\)C\. R\. Harris, K\. J\. Millman, S\. J\. Van Der Walt, R\. Gommers, P\. Virtanen, D\. Cournapeau, E\. Wieser, J\. Taylor, S\. Berg, N\. J\. Smith,et al\.Array programming with numpy\.nature585\(7825\),pp\. 357–362\.Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Hedströmet al\.\(2025\)A\. Hedström, S\. I\. Amoukou, T\. Bewley, S\. Mishra, and M\. VelosoTo steer or not to steer? mechanistic error reduction with abstention for language models\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=fUCPq5RvmH)Cited by:[§5](https://arxiv.org/html/2609.01878#S5.p2.1)\.
- Hegazyet al\.\(2026\)A\. Hegazy, M\. Elhoushi, and A\. AlanwarGuiding giants: lightweight controllers for weighted activation steering in LLMs\.InProceedings of the 6th Workshop on Trustworthy NLP \(TrustNLP 2026\),K\. Chang, N\. Mehrabi, S\. Krishna, A\. Das, J\. Dhamala, Y\. T\. Cao, T\. Kumarage, A\. Ramakrishna, C\. Christodoulopoulos, Y\. Wan, A\. Galystan, A\. Kumar, and R\. Gupta \(Eds\.\),San Diego, California,pp\. 584–599\.External Links:[Link](https://aclanthology.org/2026.trustnlp-main.46/),[Document](https://dx.doi.org/10.18653/v1/2026.trustnlp-main.46),ISBN 979\-8\-89176\-418\-7Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p6.1),[§4\.1](https://arxiv.org/html/2609.01878#S4.SS1.p2.1)\.
- Hunter \(2007\)J\. D\. HunterMatplotlib: a 2d graphics environment\.Computing in Science & Engineering9\.External Links:[Link](https://api.semanticscholar.org/CorpusID:37016120)Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Leeet al\.\(2025\)B\. W\. Lee, I\. Padhi, K\. N\. Ramamurthy, E\. Miehling, P\. Dognin, M\. Nagireddy, and A\. DhurandharProgramming refusal with conditional activation steering\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Oi47wc10sm)Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p2.1),[§2](https://arxiv.org/html/2609.01878#S2.p3.2),[§5](https://arxiv.org/html/2609.01878#S5.p2.1)\.
- Lhoestet al\.\(2021\)Q\. Lhoest, A\. Villanova del Moral, Y\. Jernite, A\. Thakur, P\. von Platen, S\. Patil, J\. Chaumond, M\. Drame, J\. Plu, L\. Tunstall, J\. Davison, M\. Šaško, G\. Chhablani, B\. Malik, S\. Brandeis, T\. Le Scao, V\. Sanh, C\. Xu, N\. Patry, A\. McMillan\-Major, P\. Schmid, S\. Gugger, C\. Delangue, T\. Matussière, L\. Debut, S\. Bekman, P\. Cistac, T\. Goehringer, V\. Mustar, F\. Lagunas, A\. Rush, and T\. WolfDatasets: a community library for natural language processing\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online and Punta Cana, Dominican Republic,pp\. 175–184\.External Links:[Link](https://aclanthology.org/2021.emnlp-demo.21),2109\.02846Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Liet al\.\(2023\)K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. WattenbergInference\-time intervention: eliciting truthful answers from a language model\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=aLLuYpn83y)Cited by:[§5](https://arxiv.org/html/2609.01878#S5.p1.1)\.
- Liet al\.\(2025\)Y\. Li, Z\. Fan, R\. Chen, X\. Gai, L\. Gong, Y\. Zhang, and Z\. LiuFairSteer: inference time debiasing for LLMs with dynamic activation steering\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11293–11312\.External Links:[Link](https://aclanthology.org/2025.findings-acl.589/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.589),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p2.1)\.
- Mann and Whitney \(1947\)H\. B\. Mann and D\. R\. WhitneyOn a Test of Whether one of Two Random Variables is Stochastically Larger than the Other\.The Annals of Mathematical Statistics18\(1\),pp\. 50 – 60\.External Links:[Document](https://dx.doi.org/10.1214/aoms/1177730491),[Link](https://doi.org/10.1214/aoms/1177730491)Cited by:[§3](https://arxiv.org/html/2609.01878#S3.p2.1)\.
- Mason and Graham \(2002\)S\. J\. Mason and N\. E\. GrahamAreas beneath the relative operating characteristics \(roc\) and relative operating levels \(rol\) curves: statistical significance and interpretation\.Quarterly Journal of the Royal Meteorological Society128\(584\),pp\. 2145–2166\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1256/003590002320603584),[Link](https://rmets.onlinelibrary.wiley.com/doi/abs/10.1256/003590002320603584),https://rmets\.onlinelibrary\.wiley\.com/doi/pdf/10\.1256/003590002320603584Cited by:[§3](https://arxiv.org/html/2609.01878#S3.p2.1)\.
- Nanda and Bloom \(2022\)N\. Nanda and J\. BloomTransformerLens\.Note:[https://github\.com/TransformerLensOrg/TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- OpenRouter \(2025\)OpenRouterOpenRouter: a unified interface for LLMs\.Note:[https://openrouter\.ai](https://openrouter.ai/)Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Kopf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. ChintalaPyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf)Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Pedregosaet al\.\(2011\)F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg,et al\.Scikit\-learn: machine learning in python\.the Journal of machine Learning research12,pp\. 2825–2830\.Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Rimskyet al\.\(2024\)N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. TurnerSteering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15504–15522\.External Links:[Link](https://aclanthology.org/2024.acl-long.828/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.828)Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p1.1),[§2](https://arxiv.org/html/2609.01878#S2.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p1.1)\.
- Rodriguezet al\.\(2025a\)P\. Rodriguez, A\. Blaas, M\. Klein, L\. Zappella, N\. Apostoloff, marco cuturi, and X\. SuauControlling language and diffusion models by transporting activations\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=l2zFn6TIQi)Cited by:[Appendix B](https://arxiv.org/html/2609.01878#A2.p4.1),[Appendix F](https://arxiv.org/html/2609.01878#A6.p1.1),[§1](https://arxiv.org/html/2609.01878#S1.p1.1),[§4\.2](https://arxiv.org/html/2609.01878#S4.SS2.p2.1),[§4\.3](https://arxiv.org/html/2609.01878#S4.SS3.p1.1),[§5](https://arxiv.org/html/2609.01878#S5.p1.1)\.
- Rodriguezet al\.\(2025b\)P\. Rodriguez, M\. Klein, E\. Gualdoni, V\. Maiorca, A\. Blaas, L\. Zappella, marco cuturi, and X\. SuauLinEAS: end\-to\-end learning of activation steering with a distributional loss\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=EBONa3tT3K)Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p1.1),[§4\.2](https://arxiv.org/html/2609.01878#S4.SS2.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p1.1)\.
- Scarliniet al\.\(2019\)B\. Scarlini, T\. Pasini, and R\. NavigliJust “OneSeC” for producing multilingual sense\-annotated data\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 699–709\.External Links:[Link](https://aclanthology.org/P19-1069/),[Document](https://dx.doi.org/10.18653/v1/P19-1069)Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p1.1),[§1](https://arxiv.org/html/2609.01878#S1.p6.1),[§4\.1](https://arxiv.org/html/2609.01878#S4.SS1.p1.1),[§4\.3](https://arxiv.org/html/2609.01878#S4.SS3.p1.1)\.
- Suauet al\.\(2024\)X\. Suau, P\. Delobelle, K\. Metcalf, A\. Joulin, N\. Apostoloff, L\. Zappella, and P\. RodriguezWhispering experts: neural interventions for toxicity mitigation in language models\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=2P6GVfSrfZ)Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p1.1),[§4\.2](https://arxiv.org/html/2609.01878#S4.SS2.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p3.1)\.
- Teamet al\.\(2025\)G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. HussenotGemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§4\.1](https://arxiv.org/html/2609.01878#S4.SS1.p1.1)\.
- Turneret al\.\(2025\)A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmidSteering language models with activation engineering\.External Links:[Link](https://openreview.net/forum?id=2XBPdPIcFK)Cited by:[§5](https://arxiv.org/html/2609.01878#S5.p1.1)\.
- Virtanenet al\.\(2020\)P\. Virtanen, R\. Gommers, T\. E\. Oliphant, M\. Haberland, T\. Reddy, D\. Cournapeau, E\. Burovski, P\. Peterson, W\. Weckesser, J\. Bright,et al\.SciPy 1\.0: fundamental algorithms for scientific computing in python\.Nature methods17\(3\),pp\. 261–272\.Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Wanget al\.\(2025a\)T\. Wang, X\. Jiao, Y\. Zhu, Z\. Chen, Y\. He, X\. Chu, J\. Gao, Y\. Wang, and L\. MaAdaptive activation steering: a tuning\-free LLM truthfulness improvement method for diverse hallucinations categories\.InTHE WEB CONFERENCE 2025,External Links:[Link](https://openreview.net/forum?id=NBHOdQJ1VE)Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p2.1),[§5](https://arxiv.org/html/2609.01878#S5.p2.1)\.
- Wanget al\.\(2025b\)W\. Wang, J\. YANG, and W\. PengSemantics\-adaptive activation intervention for LLMs via dynamic steering vectors\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8WQ7VTfPTl)Cited by:[§5](https://arxiv.org/html/2609.01878#S5.p3.1)\.
- \[32\]Wikimedia FoundationWikimedia downloads\(Website\)External Links:[Link](https://dumps.wikimedia.org/)Cited by:[§1](https://arxiv.org/html/2609.01878#S1.p6.1),[§4\.1](https://arxiv.org/html/2609.01878#S4.SS1.p2.1)\.
- Wolfet al\.\(2020\)T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. RushTransformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.emnlp-demos.6/)Cited by:[Appendix F](https://arxiv.org/html/2609.01878#A6.p3.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2609.01878#S4.SS1.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, Z\. Kolter, and D\. HendrycksRepresentation engineering: a top\-down approach to ai transparency\.External Links:2310\.01405Cited by:[§5](https://arxiv.org/html/2609.01878#S5.p1.1)\.

Table 2:WordNet definitions supplied to the LLM judge for the seven OneSeC concepts\.## Appendix AImplementation of the Posterior Gate

For numerical stability, we evaluate the posterior of Eq\.[4](https://arxiv.org/html/2609.01878#S3.E4), with the Gaussian densities of Eq\.[5](https://arxiv.org/html/2609.01878#S3.E5), entirely in log space\.

Concept\-conditional log\-densities\.Substituting the Gaussian model of Eq\.[5](https://arxiv.org/html/2609.01878#S3.E5), i\.e\.,p\(xd∣c\)=\(2π\(σdc\)2\)−1/2exp\(−\(xd−μdc\)22​\(σdc\)2\)p\\bigl\(x\_\{d\}\\mid c\\bigr\)=\\bigl\(2\\pi\(\\sigma^\{c\}\_\{d\}\)^\{2\}\\bigr\)^\{\-1/2\}\\exp\\\!\\left\(\-\\frac\{\(x\_\{d\}\-\\mu^\{c\}\_\{d\}\)^\{2\}\}\{2\(\\sigma^\{c\}\_\{d\}\)^\{2\}\}\\right\), the log\-density of neuronddunder conceptccat the observed activationxd=ht,dx\_\{d\}=h\_\{t,d\}is

Ldc:=\\displaystyle L^\{c\}\_\{d\}\\;:=log⁡p⁡\(xd∣c\)\\displaystyle\\log p\\bigl\(x\_\{d\}\\mid c\\bigr\)\(9\)=\\displaystyle=−\(xd−μdc\)22​\(σdc\)2−12​log⁡\(2​π​\(σdc\)2\)\.\\displaystyle\-\\,\\frac\{\\bigl\(x\_\{d\}\-\\mu^\{c\}\_\{d\}\\bigr\)^\{2\}\}\{2\\,\(\\sigma^\{c\}\_\{d\}\)^\{2\}\}\\;\-\\;\\frac\{1\}\{2\}\\log\\\!\\bigl\(2\\pi\\,\(\\sigma^\{c\}\_\{d\}\)^\{2\}\\bigr\)\.
The normalization term12​log⁡\(2​π​\(σdc\)2\)\\tfrac\{1\}\{2\}\\log\\bigl\(2\\pi\(\\sigma^\{c\}\_\{d\}\)^\{2\}\\bigr\)is precomputed offline once per neuron and concept, so the online cost of Eq\.[9](https://arxiv.org/html/2609.01878#A1.E9)is one squared difference and one division per neuron\.

Log\-posterior\.Taking the logarithm of Eq\.[4](https://arxiv.org/html/2609.01878#S3.E4)and substituting Eq\.[9](https://arxiv.org/html/2609.01878#A1.E9), the log\-posterior of the source concept is

log⁡p⁡\(csrc∣xd\)=Ldsrc−log⁡\(eLdsrc\+eLdtar\),\\log p\\bigl\(c^\{\\mathrm\{src\}\}\\mid x\_\{d\}\\bigr\)\\;=\\;L^\{\\mathrm\{src\}\}\_\{d\}\\;\-\\;\\log\\\!\\bigl\(e^\{L^\{\\mathrm\{src\}\}\_\{d\}\}\+e^\{L^\{\\mathrm\{tar\}\}\_\{d\}\}\\bigr\),
The denominator is evaluated with a numerically stable log\-sum\-exp\.

Computational cost\.The posterior gate adds no trained components and no additional forward passes; its entire cost is a fixed number of elementwise operations per neuron\. Offline, the per\-neuron statistics\{μdc,\(σdc\)2\}\\\{\\mu^\{c\}\_\{d\},\(\\sigma^\{c\}\_\{d\}\)^\{2\}\\\}of Eq\.[5](https://arxiv.org/html/2609.01878#S3.E5)are estimated in a single pass over the same contrastive activations already used to construct the steering vector𝐯\\mathbf\{v\}, and the memory footprint at generation time is sixDD\-dimensional vectors \(a mean, a variance, and a precomputed normalization term per concept\), i\.e\.O⁡\(D\)O\(D\)\. Online, evaluating the gate at a generation step amounts to two evaluations of Eq\.[9](https://arxiv.org/html/2609.01878#A1.E9), one log\-sum\-exp, one exponentiation, and one comparison per neuron, a constant number of elementwise operations, computed for allDDneurons simultaneously as vectorised tensor operations\. The per\-token cost is thereforeO⁡\(D\)O\(D\), the same order as the steering\-vector additionα​𝐯\\alpha\\mathbf\{v\}itself and as the token\-level gates it composes with: dimension\-level conditioning adds only a constant\-factor overhead to conditional steering, with no dependence on the number of contrastive examples at inference time\.

## Appendix BConcept Classification

To determine whether a generated continuation refers to the source concept, we use an LLM\-based binary classifier\. For each of the seven OneSeC concepts—*football*,*cloud*,*baby*,*church*,*book*,*flower*, and*balloon*—we provide the classifier with the corresponding WordNet definition and the generated continuation\. Supplying the definition allows the classifier to identify semantic references to the concept even when the concept word itself does not appear explicitly\.

The classifier is instructed to return onlyyesorno\. We use greedy decoding with temperature00and allow at most five output tokens\. The exact prompt is:

> You are a chatbot who answers whether the provided sentence is referring to \{concept\} defined as \{wordnet\_def\} Note that the sentence might not contain the word \{concept\}, but may just be referencing concept as defined\. Answer only with ’yes’ or ’no’\. \{generated\_text\}

The WordNet definitions used for the seven concepts are shown in Table[2](https://arxiv.org/html/2609.01878#A0.T2)\. Our prompt is adapted from[Rodriguez et al\. \(2025a\)](https://arxiv.org/html/2609.01878#bib.bib9)\.

Figure 4:Toxicity mitigation trade\-off curves on MMLU\. MMLU correct option probability \(capability, higher is better\) versus toxicity score \(lower = less toxic\) for Gemma\-3 \(4B, top\) and Qwen\-3 \(1\.7B, bottom\), obtained by sweeping the steering strengthα\\alpha\. Each panel compares unconditional steering \(dashed\), the token\-level conditional baseline \(CAST, left; DSAS, right\), and the same baseline augmented with our dimension\-level gates \(\+GAPS\)\. Curves closer to the top\-left corner are better; the diamond marks the unsteered model\. On Gemma\-3 \(4B\), the dimension\-gated variants retain higher MMLU performance than their token\-level counterparts at matched toxicity levels, degrading more gracefully as the intervention strength increases; on Qwen\-3 \(1\.7B\), all methods perform comparably across most of the toxicity range\.
## Appendix CMMLU Results \- Toxicity

Figure[4](https://arxiv.org/html/2609.01878#A2.F4)reports MMLU correct\-option probability against the toxicity score for both models and both steering families\.

On Gemma\-3 \(4B\), the dimension\-gated variants \(\+GAPS\) retain higher MMLU than their token\-level counterparts at matched toxicity levels, with the largest margin in the DSAS\. Whereas CAST and DSAS degrade noticeably as the intervention strengthα\\alphaincreases, the GAPS variants remain close to the unsteered baseline across most of theα\\alpharange and degrade more gracefully at the strongest interventions, reaching low toxicity scores at a substantially smaller MMLU cost\.

On Qwen\-3 \(1\.7B\), all methods perform comparably across most of the toxicity range\. At moderate intervention strengths, unconditional steering and the token\-level baselines slightly exceed the unsteered MMLU value\. At largeα\\alpha, however, their performance drops steeply, while the GAPS variants degrade more slowly at high intervention strengths\.

Figure 5:Concept removal trade\-off curves on MMLU\. MMLU correct\-option probability \(capability, higher is better\) versus average normalized concept score \(each concept’s score divided by its unsteered value, then averaged over the seven OneSeC concepts; lower = more concept removal\), obtained by sweeping the steering strengthα\\alpha, for Gemma\-3 \(4B, top\) and Qwen\-3 \(1\.7B, bottom\)\. Each panel compares unconditional steering \(dashed\), the token\-level conditional baseline \(CAST, left; DSAS, right\), and the same baseline augmented with our dimension\-level gates \(\+GAPS\)\. Curves closer to the top\-left corner are better; the diamond marks the unsteered model\. Shaded bands show±1\\pm 1standard error of the mean across the seven concepts\. On Gemma\-3 \(4B\), the dimension\-gated variants perform nearly identically to their token\-level counterparts, while on Qwen\-3 \(1\.7B\) they retain clearly higher MMLU at matched removal levels for low to moderate intervention strengths\.
## Appendix DMMLU Results \- OneSeC

Figure[5](https://arxiv.org/html/2609.01878#A3.F5)reports MMLU correct\-option probability against the average normalized concept score for both models and both token\-level gating methods\.

On Gemma\-3 \(4B\), both token\-level baselines \(CAST and DSAS\) improve over unconditional steering, which degrades steeply once the intervention strengthα\\alphabecomes large, while the conditional methods remain near the unsteered MMLU value across most of the removal range\. Because the token\-level gates already leave little capability on the table, adding our dimension\-level gates yields no further gain here: the \+GAPS variants perform nearly identically to their token\-level counterparts\.

On Qwen\-3 \(1\.7B\), the picture differs\. Token\-level gating alone is less effective: CAST, in particular, falls below unconditional steering at strong intervention strengths\. Adding the dimension\-level gates improves both families, with CAST\+GAPS and DSAS\+GAPS retaining clearly higher MMLU than their token\-level counterparts at matched removal levels for low to moderate intervention strengths\. At the strongest interventions, CAST\+GAPS also declines slightly; since the token\-level gate is applied before our dimension\-level gates, the GAPS variants inherit part of the degradation introduced by the underlying CAST gate and cannot fully recover the unconditional\-steering performance in this regime\.

Figure 6:Toxicity mitigation trade\-off curves with interventions applied at layer 18\. Setup as in Figure[2](https://arxiv.org/html/2609.01878#S3.F2), but steering at layer 18 rather than layer 15\. Our dimension\-level gates again shift the Pareto frontier toward the bottom\-left for both models and both steering families, indicating that the gains are robust to the choice of intervention layer\.
## Appendix ERobustness to the Intervention Layer

To verify that the benefits of dimension\-level gating are not specific to a single intervention site, we repeat the toxicity mitigation experiment of Section[4\.2](https://arxiv.org/html/2609.01878#S4.SS2)with all interventions applied at layer 18 instead of layer 15, keeping every other component of the protocol unchanged\. \(contrastive data, steering vectors, gate statistics, and evaluation setup are all recomputed at the new layer\)\. Figure[6](https://arxiv.org/html/2609.01878#A4.F6)reports the resulting perplexity–toxicity trade\-off curves for both models and both token\-level methods\.

The picture closely mirrors the layer\-15 results of Figure[2](https://arxiv.org/html/2609.01878#S3.F2)\. Token\-level conditional steering \(CAST, DSAS\) again improves substantially over unconditional steering, and adding our dimension\-level gates \(\+GAPS\) shifts the Pareto frontier further toward the bottom\-left corner: at matched toxicity levels, GAPS attains equal or lower perplexity than their token\-level counterparts, across both Gemma\-3 \(4B\) and Qwen\-3 \(1\.7B\) and both token\-level methods\.

The same qualitative ordering \(unconditional<<token\-level<<token\-level\+GAPS\) holds at a different layer, indicating that the gains from dimension\-level conditioning are not an artifact of a particular intervention site\. This is expected under our formulation: both gates are estimated directly from the contrastive activations of whichever layer is steered, so the separability statistics and concept\-conditional Gaussians adapt automatically to the chosen layer\. We conclude that the proposed GAPS is robust to the choice of intervention layer\.

## Appendix FImplementation Details

Dataset Statistics\.We report the data used to construct the steering directions and evaluate each task\. For toxicity steering, we use RealToxicityPrompts\([Gehman et al\., 2020](https://arxiv.org/html/2609.01878#bib.bib13)\), labeling prompts with toxicity scores≥0\.5\\geq 0\.5as toxic and those with scores<0\.2<0\.2as non\-toxic\. After shuffling with a fixed seed, we reserve the first 500 toxic prompts for evaluation, generating 10 completions per prompt, and use the next 5,000 examples from each class to collect activations\. These subsets are disjoint\. For concept steering, following Rodriguez et al\.\([Rodriguez et al\., 2025a](https://arxiv.org/html/2609.01878#bib.bib9)\), we construct contrastive sets for seven OneSeC concepts\([Scarlini et al\., 2019](https://arxiv.org/html/2609.01878#bib.bib15)\), using 700 concept\-positive and 700 concept\-negative sentences per concept to collect activations\. We evaluate each concept on 1,000 generated completions and measure the proportion that refer to the source concept using the classifier described in Appendix[B](https://arxiv.org/html/2609.01878#A2)\.

Model Specifications\.As described in the main text, we use the instruction\-tuned Gemma\-3 model with 4 billion parameters \(Hugging Face name:google/gemma\-3\-4b\-it\) and the Qwen\-3 model with 1\.7 billion parameters \(Hugging Face name:Qwen/Qwen3\-1\.7B\)\. The hidden\-state dimensionality, over which our dimension\-level gates \(GAPS\) operate, isD=2560D=2560for Gemma\-3 \(4B\) andD=2048D=2048for Qwen\-3 \(1\.7B\)\. All interventions are applied to the residual stream \(hook\_resid\_post\) at layer 15 \(layer 18 for the robustness analysis in Appendix[E](https://arxiv.org/html/2609.01878#A5)\)\.

Package Details\.Our implementation is built on PyTorch\([Paszke et al\., 2019](https://arxiv.org/html/2609.01878#bib.bib28)\)\. Models are loaded and hooked through the TransformerLens library\([Nanda and Bloom, 2022](https://arxiv.org/html/2609.01878#bib.bib27)\), which we use to cache residual\-stream activations and to apply all steering interventions during generation, in combination with the HuggingFace Transformers\([Wolf et al\., 2020](https://arxiv.org/html/2609.01878#bib.bib29)\)and Datasets\([Lhoest et al\., 2021](https://arxiv.org/html/2609.01878#bib.bib30)\)libraries for tokenizers and data loading\. Statistical computations use NumPy\([Harris et al\., 2020](https://arxiv.org/html/2609.01878#bib.bib31)\)and SciPy\([Virtanen et al\., 2020](https://arxiv.org/html/2609.01878#bib.bib32)\); scikit\-learn\([Pedregosa et al\., 2011](https://arxiv.org/html/2609.01878#bib.bib33)\)is used for the PCA and logistic\-regression probe of the DSAS baseline\. The LLM\-judge evaluation for concept removal \(Appendix[B](https://arxiv.org/html/2609.01878#A2)\) queries GPT\-4o\-mini accessed via the OpenRouter API[OpenRouter \(2025\)](https://arxiv.org/html/2609.01878#bib.bib35), and all figures are produced with Matplotlib\([Hunter, 2007](https://arxiv.org/html/2609.01878#bib.bib34)\)\.

Computation Details\.All experiments were conducted on a high\-performance computing \(HPC\) cluster managed via the Slurm workload manager\. We used NVIDIA A40 \(48 GB\) GPUs for activation collection, steering interventions, and downstream evaluation \(toxicity, perplexity, and MMLU\)\. Each job was allocated 1 GPU, 8 CPU cores, and 64 GB of RAM on a single node\.

Result Reliability\.Each reported number is an aggregate over a large number of stochastic generations: toxicity scores average over 500 prompts withk=10k=10sampled continuations each \(5,000 generations per operating point\), and concept scores average over 1,000 sampled completions per concept before being averaged across concepts\. Figures[3](https://arxiv.org/html/2609.01878#S3.F3)and[5](https://arxiv.org/html/2609.01878#A3.F5)additionally report±1\\pm 1standard error of the mean across the OneSeC concepts\. The capability metrics \(Wikipedia perplexity and MMLU correct\-option probability\) are deterministic, given the model, and require no aggregation over runs\.

## Appendix GSeparability Threshold

The thresholdτz\\tau\_\{z\}of Eq\.[3](https://arxiv.org/html/2609.01878#S3.E3)is the free parameter of the separability mask\. Its admissible range is bounded from below by statistical calibration: a Bonferroni\-corrected two\-sided test at family\-wise levelα0=10−3\\alpha\_\{0\}=10^\{\-3\}across allDDdimensions requiresτz≈5\.0\\tau\_\{z\}\\approx 5\.0\(D=2048D=2048, Qwen\-3\) to5\.15\.1\(D=2560D=2560, Gemma\-3\)\.

Any value above this floor, therefore, retains only dimensions with a statistically reliable class signal; where to placeτz\\tau\_\{z\}within the admissible range is an empirical question, which we settle with a sensitivity sweep\.

We sweepτz\\tau\_\{z\}on a single setting, toxicity mitigation with Gemma\-3 \(4B\) and DSAS\+GAPS at layer 15, and reuse the selected threshold unchanged in every other experiment \(both models, both gating families, both tasks\)\.

For eachτz∈\{5,6,7,8,9\}\\tau\_\{z\}\\in\\\{5,6,7,8,9\\\}, we sweep the intervention strengthα\\alphaand evaluate toxicity, Wikipedia perplexity, and MMLU exactly as in the main experiments\. We then select operating points under two protocols\. First, we use the fixed capability budget of the main results: the lowest toxicity whose capability cost stays within perplexity≤1\.05×\\leq 1\.05\\timesand MMLU≥0\.97×\\geq 0\.97\\timesthe unsteered model\. Second, to remove the dependence on any single budget choice, we repeat the selection for1515budgets \(relative perplexity increase in\{2,3,5,7\.5,10\}%\\\{2,3,5,7\.5,10\\\}\\%crossed with relative MMLU drop in\{1,3,5\}%\\\{1,3,5\\\}\\%\) and report the mean and standard deviation of the resulting toxicity rates\.

Table[3](https://arxiv.org/html/2609.01878#A7.T3)shows that toxicity is minimized in a band aroundτz∈\[7,8\]\\tau\_\{z\}\\in\[7,8\], with degradation on both sides under both protocols\. Below the band, atτz=6\\tau\_\{z\}=6, fixed\-budget toxicity rises to0\.56%0\.56\\%and the budget\-averaged toxicity to0\.77±0\.30%0\.77\\pm 0\.30\\%: a lower threshold admits weakly informative dimensions\.

Above the band, atτz=9\\tau\_\{z\}=9, toxicity rises again \(0\.62%0\.62\\%fixed\-budget,0\.69±0\.27%0\.69\\pm 0\.27\\%budget\-averaged\): a higher threshold discards dimensions that carry useful concept signal\. We fixτz=7\\tau\_\{z\}=7for all experiments in the paper; its toxicity is essentially on par withτz=8\\tau\_\{z\}=8, and the lower threshold retains more dimensions\.

Table 3:Sensitivity of DSAS\+GAPS to the separability thresholdτz\\tau\_\{z\}on a setting \(Gemma\-3 4B, toxicity, layer 15\)\. Left: operating point under the fixed capability budget of the main results \(perplexity≤\+5%\\leq\+5\\%, MMLU≥−3%\\geq\-3\\%; unsteered baseline\)\. Right: best feasible toxicity averaged over1515capability budgets\.