GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models

# GeoSteer: Geodesic Optimization for Activation Steering in Large Language Models
Source: [https://arxiv.org/html/2609.10658](https://arxiv.org/html/2609.10658)
Hao VoNgan LeAffiliation:University of Arkansas, Fayetteville, Arkansas, USAEmail:[\{cngo,haov,thile\}@uark\.edu](mailto:)

###### Abstract

Activation steering provides a lightweight way to control large language models \(LLMs\) by modifying their hidden activations at inference time\. Among these approaches, norm\-preserving steering aims to change model behavior without altering the activation norm, reducing the risk of representation collapse and degradation\. However, existing norm\-preserving methods are limited by predefined steering trajectories and by their reliance on one\-step updates, which may fail to capture the complex structure of activation distributions\. We proposeGeoSteer, an optimization\-based method for*norm\-preserving activation steering*\.GeoSteerformulates steering as a Riemannian optimization problem and updates activations through a sequence of small geodesic steps on the representation manifold\. To avoid fixed steering directions,GeoSteerlearns a nonlinear activation\-space objective that distinguishes desired from undesired activations, and uses this function to adaptively guide each steering step\. This multistep formulation yields smoother, more stable, and more consistent steering behavior while preserving the activation norm\. Across TruthfulQA, RealToxicityPrompts, and UltraFeedback benchmarks,GeoSteerconsistently improves over state\-of\-the\-art activation steering baselines\. These results suggest that norm\-preserving steering can be made more effective by replacing predefined one\-step edits with adaptive, geometry\-aware optimization\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.10658v1/images/teaser.png)Figure 1:Conceptual illustration ofGeoSteer\. Additive steering can move activations off the sphere, while existing norm\-preserving steering keeps activations on the sphere but typically follows predefined or heuristic trajectories\. In contrast,GeoSteeradaptively optimizes the steering path by decomposing the intervention into multiple geodesic updates\. This produces smoother trajectories toward the desired activation region while preserving the activation norm\.Large language models \(LLMs\) have demonstrated remarkable capabilities and have become an essential component of modern AI, supporting a wide range of applications\([Achiam et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib24);[Dubey et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib25);[Liu et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib26);[Yang et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib8)\)\. However, as LLMs continue to develop rapidly, it is increasingly important to study and control their societal impacts, including hallucination, bias, toxicity, and other harmful behaviors\. Since modern LLMs often contain hundreds of billions of parameters\([Chowdhery et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib34);[Brown et al\., 2020](https://arxiv.org/html/2609.10658#bib.bib33)\), controlling and aligning them through fine\-tuning is becoming increasingly challenging and resource\-intensive\.*This motivates the need for lightweight alternatives that can adapt model behavior without updating model parameters\.*

Activation steering, or representation engineering, offers a lightweight way to influence LLM behavior by modifying hidden states at inference time\([Rimsky et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib4);[Wehner et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib1);[Bartoszcze et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib3)\)\. A common strategy is to construct a steering vector from contrastive examples and add it to model activations during generation\([Li et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib5);[Rimsky et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib4);[Liu et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib27);[Lee et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib12)\)\. While effective in many settings, such additive edits are highly sensitive to the intervention scale\. If the intervention is too weak, the model behavior changes little; if it is too strong, the edited activation may move away from the model’s natural representation space and harm generation quality\([You et al\., 2026](https://arxiv.org/html/2609.10658#bib.bib28)\)\. This limitation has motivated recent work on more structured intervention mechanisms, including learnable and geometry\-aware approaches\([Wu et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib29);[Deng et al\., 2026](https://arxiv.org/html/2609.10658#bib.bib30)\)\. At the same time, the architecture of modern LLMs suggests a useful geometric perspective: normalization layers such as RMSNorm\([Zhang and Sennrich, 2019](https://arxiv.org/html/2609.10658#bib.bib31)\)regulate activation scale, making the direction of the hidden representation an important target for controlled steering\.

Norm\-preserving steering methods have therefore been proposed as a way to control model behavior while maintaining the activation norm\([Pham and Nguyen, 2024](https://arxiv.org/html/2609.10658#bib.bib6);[You et al\., 2026](https://arxiv.org/html/2609.10658#bib.bib28);[Vu and Nguyen, 2026](https://arxiv.org/html/2609.10658#bib.bib32)\)\. However, existing norm\-preserving methods still have important limitations, as conceptually illustrated in Figure[1](https://arxiv.org/html/2609.10658#S1.F1)\. While they avoid the unconstrained Euclidean shifts used by additive steering, they often follow predefined or heuristic trajectories rather than adaptively optimizing the steering path\. For example, one\-step approaches such as Angular Steering\([Vu and Nguyen, 2026](https://arxiv.org/html/2609.10658#bib.bib32)\)and Spherical Steering\([You et al\., 2026](https://arxiv.org/html/2609.10658#bib.bib28)\)typically rely on simple statistical features, such as mean activation differences, while ignoring richer distributional structure and interactions among activation dimensions\. Their reliance on one\-step updates may also fail to capture the complex patterns of activation distributions, limiting their expressive power when steering nuanced model behaviors\.

Other methods, such as HPR[Pham and Nguyen \(2024\)](https://arxiv.org/html/2609.10658#bib.bib6), introduce more complex training procedures involving neural networks, but still assume that the correct steering direction lies within a predefined two\-dimensional plane\. This geometric restriction can limit flexibility, especially when the desired behavioral change requires movement along more complex directions in representation space\. Moreover, existing methods are often sensitive to hyperparameters and may not generalize consistently across different models or tasks\.

To address these limitations, we proposeGeoSteer, an optimization\-based framework for norm\-preserving activation steering\. We observe that norm\-preserving steering can be naturally viewed as moving an activation point on a Riemannian manifold from an undesired region toward a desired region, while keeping its norm fixed\. Rather than applying a single large update or following a predefined rotation,GeoSteerdecomposes steering into a sequence of small geodesic updates[Absil et al\. \(2008\)](https://arxiv.org/html/2609.10658#bib.bib35)\. Each update is guided by the current activation state, and together these steps form a smooth, adaptive trajectory on the manifold\. We formulate this process as a Riemannian optimization problem[Absil et al\. \(2008\)](https://arxiv.org/html/2609.10658#bib.bib35)\. Specifically, we define a nonlinear activation\-space objective that distinguishes desired from undesired activations\. This objective provides an adaptive signal for selecting steering directions that increase alignment with desired activations while moving away from undesired ones\. At inference time,GeoSteerperforms iterative geodesic updates guided by this objective, ensuring that activations remain on the norm\-preserving manifold throughout the steering process\. By replacing predefined one\-step rotations with adaptive multistep optimization,GeoSteercaptures richer and more fine\-grained structure in the activation space\. This leads to more stable and expressive steering while preserving activation norms\. Experiments on TruthfulQA[Lin et al\. \(2021\)](https://arxiv.org/html/2609.10658#bib.bib21), UltraFeedback[Cui et al\. \(2023\)](https://arxiv.org/html/2609.10658#bib.bib11), and RealToxicityPrompts[Gehman et al\. \(2020\)](https://arxiv.org/html/2609.10658#bib.bib22)show thatGeoSteerconsistently outperforms state\-of\-the\-art activation steering baselines\.

Our contributions are summarized as follows:

- •We formulate norm\-preserving activation steering as Riemannian optimization on the activation sphere, providing a principled geometric view of activation intervention\.
- •We proposeGeoSteer, an adaptive multistep steering method that replaces fixed one\-step transformations with a sequence of small geodesic updates guided by the current activation state\.
- •We conduct experiments on TruthfulQA, UltraFeedback, and RealToxicityPrompts, showing thatGeoSteerconsistently improves over strong activation steering baselines while maintaining efficient inference\.

## 2Related Work

Activation Steering\.Activation steering controls LLM behavior by modifying hidden activations at inference time\. Most existing methods follow an additive steering paradigm, where a direction vector is estimated from contrastive activation patterns and added to the model’s hidden states during decoding\. The intervention typically takes the form of shifting an activation by a scaled steering vector, so that the edited representation is moved toward a desired behavioral direction in activation space\. Fixed\-vector methods such as ITI[Li et al\. \(2023\)](https://arxiv.org/html/2609.10658#bib.bib5), RepE[Zou et al\. \(2023\)](https://arxiv.org/html/2609.10658#bib.bib10), and CAA[Rimsky et al\. \(2024\)](https://arxiv.org/html/2609.10658#bib.bib4)apply the same update across inputs, which limits their adaptability\. MiMiC[Singh et al\. \(2024\)](https://arxiv.org/html/2609.10658#bib.bib7)and Linear\-AcT[Rodriguez et al\. \(2025\)](https://arxiv.org/html/2609.10658#bib.bib14)introduce more structured linear formulations, while ODESteer[Zhao et al\. \(2026\)](https://arxiv.org/html/2609.10658#bib.bib36)further improves adaptivity by treating steering as an ODE\-based trajectory\.*Nevertheless, these approaches do not explicitly constrain the activation norm, so their updates may still push representations away from the model’s natural activation geometry and degrade utility\.*

Norm\-Preserving Activation Steering\.Norm\-preserving activation steering addresses this issue by modifying activation directions while keeping activation norms unchanged\. These methods are motivated by the observation that modern LLM activations often lie on structured manifolds, where changing the direction can influence model behavior without disrupting activation magnitude\. Angular Steering[Vu and Nguyen \(2026\)](https://arxiv.org/html/2609.10658#bib.bib32)and HPR[Pham and Nguyen \(2024\)](https://arxiv.org/html/2609.10658#bib.bib6)perform geometric updates using fixed low\-dimensional planes or Householder reflection, but they still impose strong assumptions on the steering direction\. Spherical Steering[You et al\. \(2026\)](https://arxiv.org/html/2609.10658#bib.bib28)rotates activations on the sphere, but often relies on simple mean\-based statistics and ignores richer distributional structure\. Moreover, most norm\-preserving methods perform only a one\-step update, limiting their ability to capture complex activation patterns\.*In contrast, our method performs adaptive multistep updates on the manifold, recomputing the steering direction at each step based on the current activation\.*

## 3Methodology

### 3\.1Preliminaries

Activation Steering\.Activation steering aims to modify an internal hidden representation of a pretrained language model so that the model behavior shifts toward a desired attribute while preserving the original model parameters\. Leth∈ℝdh\\in\\mathbb\{R\}^\{d\}denote the activation at a selected layer and token position\. Conventional steering methods often edit the activation by adding a direction vector,

h^=h\+α​v,\\hat\{h\}=h\+\\alpha v,\(1\)wherevvis a steering direction andα\\alphacontrols the steering strength\. While simple, this additive update may change both the direction and norm of the activation, which can move the representation away from the geometry learned by the model\.

Norm\-Preserving Steering on the Hypersphere\.Norm\-preserving steering constrains the edited activation to remain on a hypersphere with the same norm as the original activation\. Given an activationhh, its normalized representation can be written as

z=h‖h‖2,z∈𝕊d−1\.z=\\frac\{h\}\{\\\|h\\\|\_\{2\}\},\\quad z\\in\\mathbb\{S\}^\{d\-1\}\.\(2\)Steering can then be viewed as modifyingzzon the unit sphere to obtain an edited normalized activationz^∈𝕊d−1\\hat\{z\}\\in\\mathbb\{S\}^\{d\-1\}\. The final edited activation in the original activation space is recovered by restoring the original norm:

h^=‖h‖2​z^\.\\hat\{h\}=\\\|h\\\|\_\{2\}\\hat\{z\}\.\(3\)Here,z^\\hat\{z\}denotes the steered direction on the unit sphere, whileh^\\hat\{h\}denotes the final edited activation used by the model\. This formulation preserves the activation norm and treats steering as a directional update over the sphere rather than as an unconstrained Euclidean shift\.

Riemannian Optimization on the Sphere\.Steering on the hypersphere can be viewed as a constrained optimization problem\. Given a normalized activationz∈𝕊d−1z\\in\\mathbb\{S\}^\{d\-1\}and an objective functionℒ⁡\(z\)\\mathcal\{L\}\(z\), the goal is to updatezzso as to improve the objective while maintaining the constraint

A standard Euclidean gradient step does not generally satisfy this constraint, since it may move the activation away from the sphere\. Riemannian optimization addresses this by computing the update direction in the tangent space of the manifold\.

For the unit sphere, the tangent space atzzis

Tz​𝕊d−1=\{u∈ℝd:u⊤​z=0\}\.T\_\{z\}\\mathbb\{S\}^\{d\-1\}=\\\{u\\in\\mathbb\{R\}^\{d\}:u^\{\\top\}z=0\\\}\.\(5\)This space contains all directions that are locally valid on the sphere\. Given the Euclidean gradient

g=∇zℒ​\(z\),g=\\nabla\_\{z\}\\mathcal\{L\}\(z\),\(6\)we obtain the Riemannian gradient by removing the component ofggthat points in the radial direction:

gℛ=g−\(g⊤​z\)​z\.g\_\{\\mathcal\{R\}\}=g\-\(g^\{\\top\}z\)z\.\(7\)The resulting vectorgℛ∈Tz​𝕊d−1g\_\{\\mathcal\{R\}\}\\in T\_\{z\}\\mathbb\{S\}^\{d\-1\}gives the steepest local change of the objective under the spherical constraint\. After choosing a tangent update direction, the point can be moved along the sphere using a geodesic update\. This provides the geometric basis forGeoSteer, whereℒ⁡\(z\)\\mathcal\{L\}\(z\)is instantiated as a learned steering objective\.

### 3\.2GeoSteer

We consider inference\-time activation steering for a pretrained language model\. Given an input promptxx, leth∈ℝdh\\in\\mathbb\{R\}^\{d\}denote the activation at a selected layer and token position\.GeoSteeredits this activation by optimizing its normalized direction on the unit sphere\. We denote the inference\-time trajectory by\{z\(t\)\}t=0K\\\{z^\{\(t\)\}\\\}\_\{t=0\}^\{K\}, initialized as

z\(0\)=h‖h‖2\.z^\{\(0\)\}=\\frac\{h\}\{\\\|h\\\|\_\{2\}\}\.\(8\)AfterKKgeodesic steps, the edited activation is reconstructed as

h^=‖h‖2​z\(K\),\\hat\{h\}=\\\|h\\\|\_\{2\}z^\{\(K\)\},\(9\)so the original activation norm is preserved\.

Learning the Steering Objective\.GeoSteerfirst learns a probe that distinguishes desired from undesired activations\. Given labeled activations

𝒟=\{\(hi,yi\)\}i=1N,\\mathcal\{D\}=\\\{\(h\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\},\(10\)whereyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}denotes the undesired or desired class, we normalize each training activation as

zi=hi‖hi‖2\.z\_\{i\}=\\frac\{h\_\{i\}\}\{\\\|h\_\{i\}\\\|\_\{2\}\}\.\(11\)The probe score is defined as:

sϕ​\(z\)=w⊤​ϕ​\(z\)\+b,s\_\{\\phi\}\(z\)=w^\{\\top\}\\phi\(z\)\+b,\(12\)whereϕ⁡\(⋅\)\\phi\(\\cdot\)is a nonlinear feature map, andwwandbbare learnable parameters\. The desired\-class probability is

pϕ​\(z\)=σ⁡\(sϕ​\(z\)\)\.p\_\{\\phi\}\(z\)=\\sigma\(s\_\{\\phi\}\(z\)\)\.\(13\)The probe is trained with the binary classification loss

ℒp​r​o​b​e=−∑i\[yilogpϕ\(zi\)\+\(1−yi\)log⁡\(1−pϕ​\(zi\)\)\]\.\\mathcal\{L\}\_\{probe\}=\-\\sum\_\{i\}\\left\[y\_\{i\}\\log p\_\{\\phi\}\(z\_\{i\}\)\+\(1\-y\_\{i\}\)\\log\(1\-p\_\\phi\(z\_i\)\)\\right\]\.

\(14\)
In our implementation,ϕ⁡\(⋅\)\\phi\(\\cdot\)is instantiated with Polynomial Count Sketch \(PCS\)[Pham and Pagh \(2013\)](https://arxiv.org/html/2609.10658#bib.bib13):

ϕ⁡\(z\)=PCS⁡\(z\)\.\\phi\(z\)=\\mathrm\{PCS\}\(z\)\.\(15\)This provides an efficient nonlinear representation without explicitly expanding high\-dimensional polynomial features\. After training, the probe is frozen and used to define the inference\-time steering objective\.

Geodesic Steering\.At inference time,GeoSteersteers the activation by maximizing the desired\-class probability under the learned probe\. Equivalently, we minimize

ℒs​t​e​e​r​\(z\)=−log⁡pϕ​\(z\)\.\\mathcal\{L\}\_\{steer\}\(z\)=\-\\log p\_\{\\phi\}\(z\)\.\(16\)Starting fromz\(0\)z^\{\(0\)\},GeoSteerperformsKKgeodesic steps with step sizeη=T/K\\eta=T/K, whereTTis the total steering strength\. At each steptt, we compute the Euclidean gradient

g\(t\)=∇z\(t\)ℒs​t​e​e​r​\(z\(t\)\),g^\{\(t\)\}=\\nabla\_\{z^\{\(t\)\}\}\\mathcal\{L\}\_\{steer\}\(z^\{\(t\)\}\),\(17\)and project it onto the tangent space of the sphere:

gℛ\(t\)=g\(t\)−\(\(g\(t\)\)⊤​z\(t\)\)​z\(t\)\.g\_\{\\mathcal\{R\}\}^\{\(t\)\}=g^\{\(t\)\}\-\\left\(\(g^\{\(t\)\}\)^\{\\top\}z^\{\(t\)\}\\right\)z^\{\(t\)\}\.\(18\)The normalized descent direction is

u\(t\)=−gℛ\(t\)‖gℛ\(t\)‖2\.u^\{\(t\)\}=\-\\frac\{g\_\{\\mathcal\{R\}\}^\{\(t\)\}\}\{\\\|g\_\{\\mathcal\{R\}\}^\{\(t\)\}\\\|\_\{2\}\}\.\(19\)GeoSteerthen updates the activation direction along the geodesic on the unit sphere[Absil et al\. \(2008\)](https://arxiv.org/html/2609.10658#bib.bib35):

z\(t\+1\)=cos⁡\(η\)​z\(t\)\+sin⁡\(η\)​u\(t\)\.z^\{\(t\+1\)\}=\\cos\(\\eta\)z^\{\(t\)\}\+\\sin\(\\eta\)u^\{\(t\)\}\.\(20\)Because this update follows the sphere, each intermediate point remains on𝕊d−1\\mathbb\{S\}^\{d\-1\}\. We provide the proof in Appendix[A\.4](https://arxiv.org/html/2609.10658#A1.SS4)\. AfterKKsteps, the edited activation is recovered as

h^=‖h‖2​z\(K\)\.\\hat\{h\}=\\\|h\\\|\_\{2\}z^\{\(K\)\}\.\(21\)
The full inference procedure is summarized in Algorithm[1](https://arxiv.org/html/2609.10658#alg1)\.

Algorithm 1GeoSteerInference0:Activation

hh, frozen probe

pϕp\_\{\\phi\}, total steering strength

TT, number of steps

KK
0:Steered activation

h^\\hat\{h\}
1:

z\(0\)←h/‖h‖2z^\{\(0\)\}\\leftarrow h/\\\|h\\\|\_\{2\}
2:

η←T/K\\eta\\leftarrow T/K
3:for

t=0t=0to

K−1K\-1do

4:

ℒs​t​e​e​r​\(z\(t\)\)←−log⁡pϕ​\(z\(t\)\)\\mathcal\{L\}\_\{steer\}\(z^\{\(t\)\}\)\\leftarrow\-\\log p\_\{\\phi\}\(z^\{\(t\)\}\)
5:

g\(t\)←∇z\(t\)ℒs​t​e​e​r​\(z\(t\)\)g^\{\(t\)\}\\leftarrow\\nabla\_\{z^\{\(t\)\}\}\\mathcal\{L\}\_\{steer\}\(z^\{\(t\)\}\)
6:

gℛ\(t\)←g\(t\)−\(\(g\(t\)\)⊤​z\(t\)\)​z\(t\)g\_\{\\mathcal\{R\}\}^\{\(t\)\}\\leftarrow g^\{\(t\)\}\-\\left\(\(g^\{\(t\)\}\)^\{\\top\}z^\{\(t\)\}\\right\)z^\{\(t\)\}
7:

u\(t\)←−gℛ\(t\)/∥gℛ\(t\)∥2u^\{\(t\)\}\\leftarrow\-g\_\{\\mathcal\{R\}\}^\{\(t\)\}/\\\|g\_\{\\mathcal\{R\}\}^\{\(t\)\}\\\|\_\{2\}
8:

z\(t\+1\)←cos⁡\(η\)​z\(t\)\+sin⁡\(η\)​u\(t\)z^\{\(t\+1\)\}\\leftarrow\\cos\(\\eta\)z^\{\(t\)\}\+\\sin\(\\eta\)u^\{\(t\)\}
9:endfor

10:

h^←‖h‖2​z\(K\)\\hat\{h\}\\leftarrow\\\|h\\\|\_\{2\}z^\{\(K\)\}
11:return

h^\\hat\{h\}

### 3\.3Comparison with Existing Norm\-Preserving Steering\.

GeoSteerdiffers from prior norm\-preserving steering methods in two main aspects\. First, rather than using a fixed geometric transformation, it treats activation steering as an optimization problem on the activation sphere\. Second, the steering direction is determined by the local gradient of a nonlinear probe, allowing the direction to adapt at each step\. This enablesGeoSteerto follow a curved trajectory toward the desired activation region, instead of relying on a single global direction or a fixed two\-dimensional rotation plane for all activations\.

## 4Experiments

Table 1:Comparison across Falcon\-7B, Mistral\-7B, and LLaMA3\.1\-8B on helpfulness, truthfulness, and detoxification\. For helpfulness, “Win” measures the win rate over the original model, and “RMmean\{\}\_\{\\text\{mean\}\}” / “RMP90\{\}\_\{\\text\{P90\}\}” report the mean and 90th\-percentile reward\. For truthfulness, “T×\\timesI” denotes Truthfulness×\\timesInformativeness, with “True” and “Info” shown separately\. For detoxification, “PPL” denotes perplexity\. Results are averaged over five runs\. Primary metrics are highlighted inblue; for main and selected auxiliary metrics, the best and second\-best results are marked byboldandunderline\.MethodModelHelpfulness\(Ultrafeedback\)Truthfulness\(TruthfulQA\)Detoxification\(Real Toxicity Prompts\)Win \(%\)↑\\bm\{\\uparrow\}RMmean\{\}\_\{\\text\{mean\}\}↑\\bm\{\\uparrow\}RMP90\{\}\_\{\\text\{P90\}\}↑\\bm\{\\uparrow\}T×\\timesI \(%\)↑\\bm\{\\uparrow\}True \(%\)↑\\bm\{\\uparrow\}Info \(%\)↑\\bm\{\\uparrow\}Toxic↓\\bm\{\\downarrow\}PPL↓\\bm\{\\downarrow\}Dist\-2↑\\bm\{\\uparrow\}OriginalFalcon\-7B50\.0\-12\.5\-2\.928\.431\.396\.30\.18115\.894\.4RepE49\.7\-12\.9\-2\.828\.731\.696\.20\.18716\.394\.7ITI48\.5\-12\.5\-2\.929\.432\.596\.10\.18715\.694\.5CAA54\.0\-12\.4\-1\.234\.137\.696\.00\.09317\.194\.7MiMiC45\.5\-13\.1\-3\.435\.443\.990\.10\.16716\.594\.1Linear\-AcT49\.8\-12\.8\-2\.534\.137\.796\.10\.16815\.994\.6ODESteer49\.0\-12\.8\-3\.433\.336\.296\.60\.08837\.593\.4HPR48\.9\-13\.5\-3\.534\.939\.295\.70\.09943\.192\.9Angular Steering51\.5\-12\.5\-2\.939\.443\.794\.70\.11316\.696\.0Spherical Steering51\.8\-12\.1\-2\.333\.445\.286\.70\.10119\.093\.8GeoSteer\(Ours\)57\.3\-11\.8\-2\.041\.146\.593\.60\.07117\.993\.5OriginalMistral\-7B50\.0\-11\.3\-1\.236\.740\.895\.60\.12518\.798\.9RepE36\.6\-14\.5\-3\.433\.343\.888\.90\.11639\.993\.2ITI47\.0\-11\.2\-1\.042\.648\.394\.00\.06022\.299\.0CAA53\.4\-9\.60\.653\.258\.694\.50\.04120\.899\.0MiMiC49\.0\-11\.4\-0\.840\.047\.192\.70\.10318\.998\.9Linear\-AcT51\.1\-10\.9\-0\.142\.647\.695\.00\.10219\.398\.9ODESteer53\.3\-10\.50\.751\.556\.994\.50\.03420\.299\.3HPR52\.3\-10\.10\.750\.456\.489\.40\.05843\.192\.9Angular Steering50\.4\-11\.1\-0\.751\.058\.093\.00\.04931\.199\.2Spherical Steering47\.3\-11\.9\-2\.458\.169\.986\.80\.04434\.398\.2GeoSteer\(Ours\)54\.7\-8\.90\.963\.270\.093\.00\.01539\.699\.1OriginalLLaMA3\.1\-8B50\.0\-10\.5\-0\.341\.544\.196\.70\.12418\.799\.0RepE49\.4\-10\.4\-1\.240\.643\.396\.70\.14338\.596\.4ITI48\.8\-10\.5\-0\.543\.746\.196\.90\.08719\.798\.6CAA57\.6\-8\.91\.550\.153\.097\.60\.04221\.599\.2MiMiC57\.8\-9\.21\.154\.160\.892\.50\.09819\.299\.3Linear\-AcT52\.8\-9\.90\.951\.553\.997\.10\.10318\.798\.9ODESteer59\.0\-8\.51\.667\.772\.694\.70\.03224\.698\.7HPR52\.2\-10\.00\.757\.060\.794\.00\.04036\.197\.5Angular Steering55\.7\-8\.21\.767\.171\.295\.50\.04320\.699\.1Spherical Steering52\.3\-9\.50\.659\.588\.470\.10\.028371\.595\.2GeoSteer\(Ours\)63\.2\-7\.42\.169\.473\.395\.80\.01825\.799\.4OriginalQwen2\.5\-7B50\.0\-5\.45\.263\.878\.285\.20\.09721\.299\.2RepE50\.7\-5\.45\.662\.577\.385\.10\.10921\.999\.1ITI48\.8\-5\.65\.865\.478\.986\.30\.09721\.299\.1CAA49\.8\-5\.45\.166\.380\.085\.80\.04920\.399\.3MiMiC48\.1\-6\.24\.062\.882\.179\.90\.09121\.299\.1Linear\-AcT47\.5\-5\.95\.363\.679\.383\.90\.08422\.199\.2ODESteer51\.4\-5\.45\.763\.779\.384\.10\.08721\.899\.0HPR48\.4\-5\.93\.865\.677\.984\.30\.04623\.599\.1Angular Steering46\.6\-6\.24\.766\.385\.680\.30\.04921\.499\.2Spherical Steering17\.2\-15\.8\-3\.061\.281\.778\.60\.034125\.997\.8GeoSteer\(Ours\)53\.2\-4\.66\.268\.182\.385\.20\.02922\.199\.2

In this section, we evaluateGeoSteeracross three representative activation\-steering tasks: helpfulness, truthfulness, and detoxification\. These tasks test whether the proposed method can steer model behavior toward desirable attributes while preserving generation quality\.

Base LLMs\.We conduct experiments on four open\-source language models with comparable parameter scales: Falcon\-7B\([Almazrouei et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib16)\), Mistral\-7B\-v0\.3\([Jiang et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib15)\), LLaMA\-3\.1\-8B\([Meta AI, 2024](https://arxiv.org/html/2609.10658#bib.bib9)\), and Qwen2\.5\-7B\([Team, 2024](https://arxiv.org/html/2609.10658#bib.bib23)\)\. Additional details on the model implementations are provided in Appendix[A\.5](https://arxiv.org/html/2609.10658#A1.SS5)\.

Baselines\.We compareGeoSteeragainst a diverse set of activation\-steering baselines\. These include Representation Engineering \(RepE\)\([Zou et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib10)\), Inference\-Time Intervention \(ITI\)\([Li et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib5)\), Contrastive Activation Addition \(CAA\)\([Rimsky et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib4)\), Minimally Modified Counterfactuals \(MiMiC\)\([Singh et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib7)\), Linear\-AcT\([Rodriguez et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib14)\), and ODESteer\([Zhao et al\., 2026](https://arxiv.org/html/2609.10658#bib.bib36)\)\. We also include norm\-preserving methods that are most closely related to our setting, including HPR\([Pham and Nguyen, 2024](https://arxiv.org/html/2609.10658#bib.bib6)\), Spherical Steering\([You et al\., 2026](https://arxiv.org/html/2609.10658#bib.bib28)\), and Angular Steering\([Vu and Nguyen, 2026](https://arxiv.org/html/2609.10658#bib.bib32)\)\. To ensure a controlled comparison, all methods are evaluated under the same steering protocol: we use the same steering layer and apply the intervention to newly generated tokens, following common practice in prior activation\-steering evaluations\([Wehner et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib1);[Bartoszcze et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib3)\)\. Additional details on the selected steering layers are provided in Appendix[A\.6](https://arxiv.org/html/2609.10658#A1.SS6)\.

Scope of Comparison\.We focus on methods that are directly compatible with single\-layer inference\-time activation steering\. Therefore, we do not include methods designed for substantially different goals, such as multi\-attribute control\([Nguyen et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib17)\), differential privacy\([Goel et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib19)\), or instruction\-following improvement\([Stolfo et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib18)\)\. We also exclude SADI\([Wang et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib20)\), as it requires interventions across all layers and therefore does not fit our single\-layer evaluation setup\.

Datasets and Metrics\.We evaluate on benchmarks corresponding to the three steering objectives\. For helpfulness, we use UltraFeedback\([Cui et al\., 2023](https://arxiv.org/html/2609.10658#bib.bib11)\)and report the win rate against responses from the original model as the main metric, along with mean reward and 90th\-percentile reward\. For truthfulness, we evaluate on TruthfulQA\([Lin et al\., 2021](https://arxiv.org/html/2609.10658#bib.bib21)\), using Truth×\\timesInfo as the primary metric and reporting Truth and Info separately for completeness\. For detoxification, we use RealToxicityPrompts\([Gehman et al\., 2020](https://arxiv.org/html/2609.10658#bib.bib22)\)and report toxicity as the main metric\. We additionally report perplexity and Dist\-nnto measure generation quality and diversity\. Further details on the datasets and evaluation metrics are provided in Appendix[A\.7](https://arxiv.org/html/2609.10658#A1.SS7)\.

### 4\.1Experimental Results

Table[1](https://arxiv.org/html/2609.10658#S4.T1)reports the main results across helpfulness, truthfulness, and detoxification\. Overall,GeoSteerconsistently achieves the best performance on the primary metrics across all evaluated models\. Specifically,GeoSteerobtains the highest win rate on UltraFeedback, the highest truthfulness×\\timesinformativeness score on TruthfulQA, and the lowest toxicity score on RealToxicityPrompts\. These results show thatGeoSteercan effectively steer model behavior toward desirable attributes while maintaining strong generation quality\. The improvement over additive and linear steering methods, such as CAA, MiMiC, ITI, and Linear\-AcT, suggests the importance of respecting the geometry of the activation space\. Since these methods typically apply a fixed Euclidean intervention, they may distort the activation representation or move it away from the model’s learned activation manifold\. In contrast,GeoSteerperforms steering directly on the hypersphere, preserving the activation norm throughout the update\. Compared with existing norm\-preserving methods, including HPR, Angular Steering, and Spherical Steering,GeoSteerfurther benefits from its adaptive geodesic trajectory\. Rather than assuming a fixed transformation or a single rotation path,GeoSteerrecomputes the steering direction at each intermediate point\. By dividing a large steering step into multiple smaller geodesic updates,GeoSteercan follow a more flexible path toward the desired activation region, leading to stronger and more consistent performance across models and tasks\.

### 4\.2Ablation Studies

![Refer to caption](https://arxiv.org/html/2609.10658v1/images/ablation_step.png)\(a\)Effect of the number of geodesic steps\.
![Refer to caption](https://arxiv.org/html/2609.10658v1/images/ablation_T.png)\(b\)Effect of total steering strengthTT\.

Figure 2:Ablation studies on TruthfulQA using True×\\timesInfo as the main metric\. We analyze the effect of the number of geodesic steps and the total steering strengthTT\.We conduct ablation studies to analyze the effect of three important design choices inGeoSteer: the number of geodesic steps, the total steering strength, and the objective function design\. All ablations are evaluated on TruthfulQA using True×\\timesInfo as the primary metric\.

Effect of the Number of Steps\.Figure[2\(a\)](https://arxiv.org/html/2609.10658#S4.F2.sf1)studies how the number of geodesic steps affects performance when the total steering strength is fixed\. Increasing the number of steps, or equivalently reducing the step size, brings a modest initial gain before the performance stabilizes\. This suggests that a moderate number of steps is sufficient for reliable integration, and thatGeoSteeris not sensitive to the exact step\-size choice\. The stable trend also shows that the objective function provides a reliable steering direction along the path\. Rather than relying on one large intervention,GeoSteerrepeatedly recomputes the direction at intermediate points, allowing the activation to move more smoothly on the sphere\.

Effect of Steering Strength\.Figure[2\(b\)](https://arxiv.org/html/2609.10658#S4.F2.sf2)shows the effect of the total steering strengthTT, with the number of steps fixed to 15\. Overall,GeoSteeris not overly sensitive to small changes inTT, and the performance remains strong within a reasonable range\. WhenTTis too small, the intervention is too weak to sufficiently steer the model, leading to limited gains\. In contrast, whenTTis too large, the intervention can overly perturb the activation and degrade generation quality, which in turn reduces performance\. Different models still exhibit slightly different preferred ranges\. For Mistral\-7B and LLaMA3\.1\-8B, performance improves asTTincreases from 0\.5 and remains strong aroundT=0\.64T=0\.64\. Falcon\-7B is more stable across the tested range, with smaller performance variation\. Based on these results, we setT=0\.64T=0\.64in the main experiments, as it provides a strong and stable setting across models without requiring model\-specific tuning\.

Table 2:Runtime analysis across different backbone models\. Tokens/s measures autoregressive generation speed\.MethodFalcon\-7BMistral\-7BLLaMA3\.1\-8BOriginal104\.296\.1104\.6RepE104\.095\.7102\.9ITI104\.095\.7103\.0CAA104\.195\.8103\.2MiMiC101\.093\.9100\.8Linear\-AcT104\.195\.9103\.3ODESteer103\.895\.5102\.9Angular Steering104\.195\.7103\.0Spherical Steering104\.195\.6103\.1GeoSteer\(Ours\)103\.995\.5102\.8

Table 3:Ablation study on the objective function design using LLaMA3\.1\-8B\. We compare a linear objective against two nonlinear alternatives based on Random Fourier Features \(RFF\) and Polynomial Count Sketch\.Objective FunctionTrue↑\\uparrowInfo↑\\uparrowTrue×\\timesInfo↑\\uparrowLinear66\.296\.262\.4RFF71\.496\.067\.8Polynomial Count Sketch73\.395\.869\.4

Effect of Objective Function Design\.Table[3](https://arxiv.org/html/2609.10658#S4.T3)studies how the choice of objective function affects the steering performance\. We compare a simple linear objective with two nonlinear alternatives: Random Fourier Features \(RFF\)[Rahimi and Recht \(2007\)](https://arxiv.org/html/2609.10658#bib.bib37)and Polynomial Count Sketch\. The linear objective already achieves strong information preservation, with an Info score of 96\.2, but its True score is relatively lower\. This suggests that a purely linear objective can preserve the original generation quality, but is less effective at capturing the nonlinear separation between desired and undesired activation regions\. Introducing nonlinear objectives improves the True score while maintaining a similar Info score\. RFF increases the True score from 66\.2 to 71\.4, leading to a clear improvement in the combined True×\\timesInfo metric\. Polynomial Count Sketch performs the best among the three objectives, achieving 73\.3 on True and 69\.4 on True×\\timesInfo, while keeping Info nearly unchanged\. These results indicate that modeling nonlinear relationships in the activation space provides a more effective steering direction without sacrificing informativeness\.

### 4\.3Runtime analysis

We further evaluate the computational overhead ofGeoSteerduring autoregressive generation\. The experiment measures generation throughput in tokens per second on 4 NVIDIA A100 40GB GPUs across three backbone models\. As shown in Table[2](https://arxiv.org/html/2609.10658#S4.T2), the original models achieve 104\.2, 96\.1, and 104\.6 tokens/s on Falcon\-7B, Mistral\-7B, and LLaMA3\.1\-8B, respectively\.GeoSteerachieves 103\.9, 95\.5, and 102\.8 tokens/s on the same backbones, corresponding to relative overheads of 0\.3%, 0\.6%, and 1\.7%, respectively\. On average,GeoSteerretains 100\.7 tokens/s compared with 101\.6 tokens/s for the original models, resulting in an average overhead of only 0\.9%\. This small decrease is expected becauseGeoSteerperforms iterative geodesic updates during generation\. Nevertheless, the overhead remains negligible in practice and is comparable to other efficient steering methods\. These results show thatGeoSteerpreserves almost all of the original generation throughput, making it practical for autoregressive inference\.

## 5Conclusion

In this work, we introducedGeoSteer, an optimization\-based method for norm\-preserving activation steering\. Unlike existing methods that rely on predefined one\-step edits,GeoSteerformulates steering as a Riemannian optimization problem and moves activations through a sequence of small geodesic steps on the activation sphere\. At each step, the steering direction is adaptively recomputed from the local gradient of a learned nonlinear activation\-space objective\. This design better respects the geometry of normalized representations and provides a smoother and more stable way to control model behavior at inference time\. Across TruthfulQA, RealToxicityPrompts, and UltraFeedback,GeoSteerconsistently improves over strong activation steering baselines\. Our ablation studies further show the importance of the number of geodesic steps, the total steering strength, and the nonlinear objective function design, while the runtime analysis demonstrates that the additional cost remains modest during autoregressive generation\.

## 6Limitations

AlthoughGeoSteerprovides an effective framework for norm\-preserving activation steering, it has several limitations\.

- •Integration with other direction\-discovery methods\.Our current implementation learns the steering objective from contrastive activation examples\. This makes the method simple and compatible with standard activation\-steering pipelines, but it does not directly incorporate other direction\-discovery mechanisms, such as sparse autoencoders \(SAEs\)\. ExtendingGeoSteerto SAE\-based features is non\-trivial because SAE representations are often high\-dimensional, sparse, and feature\-wise rather than direction\-wise\. It is therefore unclear how to select stable feature combinations, convert them into smooth tangent directions on the activation sphere, and perform geodesic updates efficiently without introducing noise or instability\.
- •Potential risks and unintended behavior\.Activation steering is a general\-purpose inference\-time control mechanism\. Although our experiments focus on desirable behaviors, such as improving truthfulness, reducing toxicity, and improving preference\-oriented generation, the same type of method could be misused with inappropriate objectives or steering directions to induce undesirable behaviors, including toxic, biased, deceptive, or otherwise harmful outputs\. In addition, preserving the activation norm does not guarantee that all unintended behavioral changes are avoided\. Therefore, activation steering methods should be carefully evaluated for downstream safety, robustness, and misuse risks before deployment\.

As future work, we plan to investigate stable integration ofGeoSteerwith SAE\-based feature discovery and to conduct broader safety evaluations that better characterize unintended behavioral changes and misuse risks across tasks and model families\.

## References

- Absilet al\.\(2008\)P\. Absil, R\. Mahony, and R\. SepulchreOptimization algorithms on matrix manifolds\.Princeton University Press\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p5.1),[§3\.2](https://arxiv.org/html/2609.10658#S3.SS2.p4.5)\.
- Achiamet al\.\(2023\)J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p1.1)\.
- Almazroueiet al\.\(2023\)E\. Almazrouei, H\. Alobeidli, A\. Alshamsi, A\. Cappelli, R\. Cojocaru, M\. Debbah, É\. Goffinet, D\. Hesslow, J\. Launay, Q\. Malartic,et al\.The falcon series of open language models\.arXiv preprint arXiv:2311\.16867\.Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p2.1)\.
- Bartoszczeet al\.\(2025\)L\. Bartoszcze, S\. Munshi, B\. Sukidi, J\. Yen, Z\. Yang, D\. Williams\-King, L\. Le, K\. Asuzu, and C\. MapleRepresentation engineering for large\-language models: survey and research challenges\.arXiv preprint arXiv:2502\.17601\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p1.1)\.
- Chowdheryet al\.\(2023\)A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.Palm: scaling language modeling with pathways\.Journal of machine learning research24\(240\),pp\. 1–113\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p1.1)\.
- Cuiet al\.\(2023\)G\. Cui, L\. Yuan, N\. Ding, G\. Yao, B\. He, W\. Zhu, Y\. Ni, G\. Xie, R\. Xie, Y\. Lin,et al\.Ultrafeedback: boosting language models with scaled ai feedback\.arXiv preprint arXiv:2310\.01377\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p5.1),[§4](https://arxiv.org/html/2609.10658#S4.p5.1)\.
- Denget al\.\(2026\)C\. Deng, R\. Chang, and H\. ChenSteering information utility in key\-value memory for language model post\-training\.Advances in Neural Information Processing Systems38,pp\. 102036–102064\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1)\.
- Dubeyet al\.\(2024\)A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p1.1)\.
- Gehmanet al\.\(2020\)S\. Gehman, S\. Gururangan, M\. Sap, Y\. Choi, and N\. A\. SmithRealtoxicityprompts: evaluating neural toxic degeneration in language models\.arXiv preprint arXiv:2009\.11462\.Cited by:[§A\.7](https://arxiv.org/html/2609.10658#A1.SS7.p6.1),[§1](https://arxiv.org/html/2609.10658#S1.p5.1),[§4](https://arxiv.org/html/2609.10658#S4.p5.1)\.
- Goelet al\.\(2025\)A\. Goel, Y\. Hu, I\. Gurevych, and A\. SanyalDifferentially private steering for large language model alignment\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=lLkgj7FEtZ)Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p4.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. Le Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. El SayedMistral 7b\.arXiv preprint arXiv:2310\.06825\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.06825),[Link](https://arxiv.org/abs/2310.06825)Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p2.1)\.
- Lambertet al\.\(2025\)N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. HajishirziRewardBench: evaluating reward models for language modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1755–1797\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.96/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.96),ISBN 979\-8\-89176\-195\-7Cited by:[§A\.7](https://arxiv.org/html/2609.10658#A1.SS7.p2.1)\.
- Leeet al\.\(2024\)B\. W\. Lee, I\. Padhi, K\. N\. Ramamurthy, E\. Miehling, P\. Dognin, M\. Nagireddy, and A\. DhurandharProgramming refusal with conditional activation steering\.arXiv preprint arXiv:2409\.05907\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1)\.
- Liet al\.\(2023\)K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. WattenbergInference\-time intervention: eliciting truthful answers from a language model\.Advances in Neural Information Processing Systems36,pp\. 41451–41530\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1),[§2](https://arxiv.org/html/2609.10658#S2.p1.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Linet al\.\(2021\)S\. Lin, J\. Hilton, and O\. EvansTruthfulqa: measuring how models mimic human falsehoods\.arXiv preprint arXiv:2109\.07958\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p5.1),[§4](https://arxiv.org/html/2609.10658#S4.p5.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p1.1)\.
- Liuet al\.\(2023\)S\. Liu, H\. Ye, L\. Xing, and J\. ZouIn\-context vectors: making in context learning more effective and controllable through latent space steering\.arXiv preprint arXiv:2311\.06668\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1)\.
- Meta AI \(2024\)Meta AIMeta llama 3\.1 8b model card\.Note:[https://huggingface\.co/meta\-llama/Llama\-3\.1\-8B](https://huggingface.co/meta-llama/Llama-3.1-8B)Released July 23, 2024Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p2.1)\.
- Nguyenet al\.\(2025\)D\. Nguyen, A\. Prasad, E\. Stengel\-Eskin, and M\. BansalMulti\-attribute steering of language models via targeted intervention\.arXiv preprint arXiv:2502\.12446\.Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p4.1)\.
- Pham and Pagh \(2013\)N\. Pham and R\. PaghFast and scalable polynomial kernels via explicit feature maps\.InProceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining,pp\. 239–247\.Cited by:[§A\.3](https://arxiv.org/html/2609.10658#A1.SS3.p1.1),[§3\.2](https://arxiv.org/html/2609.10658#S3.SS2.p3.1)\.
- Pham and Nguyen \(2024\)V\. Pham and T\. H\. NguyenHouseholder pseudo\-rotation: a novel approach to activation editing in llms with direction\-magnitude perspective\.arXiv preprint arXiv:2409\.10053\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p3.1),[§1](https://arxiv.org/html/2609.10658#S1.p4.1),[§2](https://arxiv.org/html/2609.10658#S2.p2.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Rahimi and Recht \(2007\)A\. Rahimi and B\. RechtRandom features for large\-scale kernel machines\.Advances in neural information processing systems20\.Cited by:[§4\.2](https://arxiv.org/html/2609.10658#S4.SS2.p4.1)\.
- Rimskyet al\.\(2024\)N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. TurnerSteering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 15504–15522\.External Links:[Link](https://aclanthology.org/2024.acl-long.828/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.828)Cited by:[§A\.6](https://arxiv.org/html/2609.10658#A1.SS6.p1.1),[§1](https://arxiv.org/html/2609.10658#S1.p2.1),[§2](https://arxiv.org/html/2609.10658#S2.p1.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Rodriguezet al\.\(2025\)P\. Rodriguez, A\. Blaas, M\. Klein, L\. Zappella, N\. Apostoloff, marco cuturi, and X\. SuauControlling language and diffusion models by transporting activations\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=l2zFn6TIQi)Cited by:[§2](https://arxiv.org/html/2609.10658#S2.p1.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Singhet al\.\(2024\)S\. Singh, S\. Ravfogel, J\. Herzig, R\. Aharoni, R\. Cotterell, and P\. KumaraguruRepresentation surgery: theory and practice of affine steering\.arXiv preprint arXiv:2402\.09631\.Cited by:[§2](https://arxiv.org/html/2609.10658#S2.p1.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Stolfoet al\.\(2025\)A\. Stolfo, V\. Balachandran, S\. Yousefi, E\. Horvitz, and B\. NushiImproving instruction\-following in language models through activation steering\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=wozhdnRCtw)Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p4.1)\.
- Team \(2024\)Q\. TeamQwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p2.1)\.
- Vu and Nguyen \(2026\)M\. H\. Vu and T\. NguyenAngular steering: behavior control via rotation in activation space\.Advances in Neural Information Processing Systems38,pp\. 121653–121690\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p3.1),[§2](https://arxiv.org/html/2609.10658#S2.p2.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Wanget al\.\(2025\)W\. Wang, J\. YANG, and W\. PengSemantics\-adaptive activation intervention for LLMs via dynamic steering vectors\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8WQ7VTfPTl)Cited by:[§4](https://arxiv.org/html/2609.10658#S4.p4.1)\.
- Wehneret al\.\(2025\)J\. Wehner, S\. Abdelnabi, D\. Tan, D\. Krueger, and M\. FritzTaxonomy, opportunities, and challenges of representation engineering for large language models\.arXiv preprint arXiv:2502\.19649\.Cited by:[§A\.7](https://arxiv.org/html/2609.10658#A1.SS7.p7.1),[§1](https://arxiv.org/html/2609.10658#S1.p2.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Wuet al\.\(2024\)Z\. Wu, A\. Arora, Z\. Wang, A\. Geiger, D\. Jurafsky, C\. D\. Manning, and C\. PottsReft: representation finetuning for language models\.Advances in Neural Information Processing Systems37,pp\. 63908–63962\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p1.1)\.
- Youet al\.\(2026\)Z\. You, C\. Deng, and H\. ChenSpherical steering: geometry\-aware activation rotation for language models\.arXiv preprint arXiv:2602\.08169\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1),[§1](https://arxiv.org/html/2609.10658#S1.p3.1),[§2](https://arxiv.org/html/2609.10658#S2.p2.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Zhang and Sennrich \(2019\)B\. Zhang and R\. SennrichRoot mean square layer normalization\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2609.10658#S1.p2.1)\.
- Zhaoet al\.\(2026\)H\. Zhao, H\. Sun, J\. Kong, X\. Li, Q\. Wang, L\. Jiang, Q\. Zhu, T\. Abdelzaher, Y\. Choi, M\. Li,et al\.Odesteer: a unified ode\-based steering framework for llm alignment\.arXiv preprint arXiv:2602\.17560\.Cited by:[§2](https://arxiv.org/html/2609.10658#S2.p1.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.
- Zouet al\.\(2023\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§2](https://arxiv.org/html/2609.10658#S2.p1.1),[§4](https://arxiv.org/html/2609.10658#S4.p3.1)\.

## Appendix AAppendix

### A\.1LLM Usage

In this work, Large Language Models \(LLMs\) were used only as writing and coding assistants\. Specifically, they were used to help polish the manuscript for grammar, clarity, and readability, and to draft small portions of experimental code\.

All LLM\-assisted content was carefully reviewed, verified, and revised by the authors\. The research ideas, theoretical framework, methodology, experimental design, and analysis were developed by the authors\. LLMs were not used to generate the core scientific contributions or to draw conclusions from the experimental results\.

The authors take full responsibility for the correctness of the theoretical claims, the validity of the experiments, and the reported results\. Any text or code produced with LLM assistance was checked to ensure correctness, originality, and compliance with ethical research standards\.

### A\.2Scientific Artifacts

Our experiments use only publicly available scientific artifacts, including benchmark datasets, pretrained language models, evaluation models, and baseline implementations\. The datasets used in our evaluation include TruthfulQA, RealToxicityPrompts, and UltraFeedback/UltraFeedback Binarized\. TruthfulQA is used for truthfulness evaluation, RealToxicityPrompts is used for toxicity\-oriented generation evaluation, and UltraFeedback is used for preference\-oriented evaluation\. These datasets are English\-language text benchmarks and are used only for research evaluation\. We report aggregate results and do not release newly collected human\-subject data\.

We use all artifacts in accordance with their respective licenses and terms of use\. TruthfulQA and RealToxicityPrompts are released under Apache\-2\.0, while UltraFeedback/UltraFeedback Binarized is released under the MIT license\. The pretrained language models used in our experiments are also publicly available: Falcon\-7B, Mistral\-7B, and Qwen2\.5\-7B are released under Apache\-2\.0, while LLaMA3\.1\-8B is released under the LLaMA 3\.1 Community License\. We use these models only as research artifacts for controlled evaluation of activation\-steering methods\.

Our use of these artifacts is consistent with their intended research and evaluation purposes\. We do not redistribute modified versions of the datasets or pretrained models as part of this work\.

We use publicly available benchmark datasets and do not collect new human\-subject data\. Some datasets used in our experiments may contain sensitive or offensive text by construction, especially RealToxicityPrompts, which is designed for evaluating toxic language generation\. We use these datasets only for research evaluation, do not redistribute modified versions of the data, and report aggregate evaluation results rather than individual examples\. We do not release any newly collected personally identifying information\.

### A\.3PCS Hyperparameters

As described in[3\.2](https://arxiv.org/html/2609.10658#S3.SS2), we use Polynomial Count Sketch\([Pham and Pagh, 2013](https://arxiv.org/html/2609.10658#bib.bib13)\)to construct randomized polynomial features\. This provides an efficient approximation to the polynomial kernel

K⁡\(x,y\)=\(γ​x⊤​y\+c0\)d,K\(x,y\)=\(\\gamma x^\{\\top\}y\+c\_\{0\}\)^\{d\},\(22\)whereγ\\gammacontrols the scale of the inner product,c0c\_\{0\}is the constant offset, anddddenotes the polynomial degree\. The approximation also depends on the number of random features, denoted byNpolyN\_\{\\text\{poly\}\}\. We setγ=0\.1\\gamma=0\.1,c0=1\.0c\_\{0\}=1\.0,d=2d=2, andNpoly=8000N\_\{\\text\{poly\}\}=8000in all experiments\. We found this configuration to be stable across different datasets and base models\.

### A\.4Norm Preservation of GeoSteer

We show that the GeoSteer update preserves the unit\-norm constraint\. At steptt, assume that the current activation direction lies on the unit sphere:

‖z\(t\)‖2=1\.\\\|z^\{\(t\)\}\\\|\_\{2\}=1\.\(23\)The update directionu\(t\)u^\{\(t\)\}is obtained by projecting the gradient onto the tangent space of the sphere and normalizing it\. Therefore,

‖u\(t\)‖2=1,\(z\(t\)\)⊤​u\(t\)=0\.\\\|u^\{\(t\)\}\\\|\_\{2\}=1,\\qquad\\left\(z^\{\(t\)\}\\right\)^\{\\top\}u^\{\(t\)\}=0\.\(24\)GeoSteer updates the activation direction as

z\(t\+1\)=cos⁡\(η\)​z\(t\)\+sin⁡\(η\)​u\(t\)\.z^\{\(t\+1\)\}=\\cos\(\\eta\)z^\{\(t\)\}\+\\sin\(\\eta\)u^\{\(t\)\}\.\(25\)To verify that the updated point remains on the sphere, we compute its squared norm:

‖z\(t\+1\)‖22\\displaystyle\\left\\\|z^\{\(t\+1\)\}\\right\\\|\_\{2\}^\{2\}=‖cos⁡\(η\)​z\(t\)\+sin⁡\(η\)​u\(t\)‖22\.\\displaystyle=\\left\\\|\\cos\(\\eta\)z^\{\(t\)\}\+\\sin\(\\eta\)u^\{\(t\)\}\\right\\\|\_\{2\}^\{2\}\.\(26\)Expanding the squared norm gives

‖z\(t\+1\)‖22\\displaystyle\\left\\\|z^\{\(t\+1\)\}\\right\\\|\_\{2\}^\{2\}=cos2⁡\(η\)​‖z\(t\)‖22\+sin2⁡\(η\)​‖u\(t\)‖22\\displaystyle=\\cos^\{2\}\(\\eta\)\\left\\\|z^\{\(t\)\}\\right\\\|\_\{2\}^\{2\}\+\\sin^\{2\}\(\\eta\)\\left\\\|u^\{\(t\)\}\\right\\\|\_\{2\}^\{2\}\(27\)\+2​cos⁡\(η\)​sin⁡\(η\)​\(z\(t\)\)⊤​u\(t\)\.\\displaystyle\+2\\cos\(\\eta\)\\sin\(\\eta\)\\left\(z^\{\(t\)\}\\right\)^\{\\top\}u^\{\(t\)\}\.Sincez\(t\)z^\{\(t\)\}andu\(t\)u^\{\(t\)\}are unit vectors and are orthogonal, we obtain

‖z\(t\+1\)‖22\\displaystyle\\left\\\|z^\{\(t\+1\)\}\\right\\\|\_\{2\}^\{2\}=cos2⁡\(η\)\+sin2⁡\(η\)\\displaystyle=\\cos^\{2\}\(\\eta\)\+\\sin^\{2\}\(\\eta\)\(28\)=1\.\\displaystyle=1\.Thus,

z\(t\+1\)∈𝕊d−1\.z^\{\(t\+1\)\}\\in\\mathbb\{S\}^\{d\-1\}\.\(29\)Since the initial pointz\(0\)=h/‖h‖2z^\{\(0\)\}=h/\\\|h\\\|\_\{2\}is on the unit sphere, the same argument applies at every step\. Therefore, all intermediate points remain on the sphere, and the final reconstructed activation

h^=‖h‖2​z\(K\)\\hat\{h\}=\\\|h\\\|\_\{2\}z^\{\(K\)\}\(30\)preserves the original activation norm\.

### A\.5Steering Model Details

We conduct experiments using the following open\-source language models:

- •Falcon\-7B:tiiuae/falcon\-7b;
- •Mistral\-7B:mistralai/Mistral\-7B\-v0\.3;
- •LLaMA3\.1\-8B:meta\-llama/Llama\-3\.1\-8B;
- •Qwen2\.5\-7B:Qwen/Qwen2\.5\-7B\.

For all models and tasks, we use the same generation configuration: temperature is set to0\.70\.7, top\-ppto0\.90\.9, and repetition penalty to1\.11\.1\.

### A\.6Steering Position

For a fair comparison, we applyGeoSteerand all baselines at the same residual\-stream position within each model\. Steering is applied to all newly generated tokens during decoding\. To choose the steering layer, we first run CAA\([Rimsky et al\., 2024](https://arxiv.org/html/2609.10658#bib.bib4)\)on TruthfulQA across all layers of each model and evaluate performance using the True×\\timesInfo metric\. The layer\-wise results are shown in Figure[3](https://arxiv.org/html/2609.10658#A1.F3)\.

Based on this analysis, we use layer 15 for Falcon\-7B, layer 16 for Mistral\-7B, layer 14 for LLaMA3\.1\-8B, and layer 14 for Qwen2\.5\-7B\. We use CAA only as a layer\-selection tool, so that all methods are evaluated under the same steering position rather than tuning the layer separately for each method\.

Figure 3:Layer\-wise steering performance on TruthfulQA using CAA for layer selection\. We report True×\\timesInfo across layers 11–18 for each base model\.
### A\.7Datasets

UltraFeedback\.We use the UltraFeedback Binarized dataset, where each prompt is associated with a preferred response and a rejected response\. From this dataset, we construct 10k training pairs, 500 validation pairs, and 500 test prompts under three random seeds\. For evaluation, we score model outputs using Skywork\-Reward\-V2\-LLaMA\-3\.1\-8B\. We report the average reward score, denoted as RMmean, the 90th percentile reward score, denoted as RMP90, and the reward\-model win rate relative to the baseline model\.

Reward\-Model Win Rate\.Given a set of prompts\{xi\}i=1N\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}and two systemsAAandBB, letsiAs\_\{i\}^\{A\}andsiBs\_\{i\}^\{B\}denote their reward\-model scores on the same prompt\. Following[Lambert et al\. \(2025\)](https://arxiv.org/html/2609.10658#bib.bib2), we define the win rate of systemAAover systemBBas:

Win\(A,B\)=1N∑i=1N\[\\displaystyle\\mathrm\{Win\}\(A,B\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\Big\[𝕀⁡\(siA\>siB\)\\displaystyle\\mathbb\{I\}\(s\_\{i\}^\{A\}\>s\_\{i\}^\{B\}\)\(31\)\+12𝕀\(siA=siB\)\]\.\\displaystyle\+\\frac\{1\}\{2\}\\mathbb\{I\}\(s\_\{i\}^\{A\}=s\_\{i\}^\{B\}\)\\Big\]\.
A win rate of0\.50\.5indicates parity with the baseline, while values above0\.50\.5indicate that systemAAis preferred by the reward model more often than systemBB\. Ties are counted as half wins\.

TruthfulQA\.For TruthfulQA, we follow the generation\-based evaluation setting used by Li et al\. \(2023\)\. The benchmark contains 817 questions, which are expanded into 5,918 question–answer pairs\. We use 40% of the data for training and 10% for validation to select hyperparameters\. We then conduct two\-fold cross validation so that all TruthfulQA questions are included in the test split\. The original TruthfulQA benchmark used two fine\-tuned GPT\-3 models as truthfulness and informativeness judges\. Since these models are no longer publicly available, we instead useallenai/truthfulqa\-truth\-judge\-llama2\-7B,allenai/truthfulqa\-info\-judge\-llama2\-7Bto evaluate truthfulness and informativeness, respectively\.

RealToxicityPrompts\.For detoxification, we use the Jigsaw Unintended Bias in Toxicity Classification dataset for training and RealToxicityPrompts[Gehman et al\. \(2020\)](https://arxiv.org/html/2609.10658#bib.bib22)for testing\. Specifically, we uniformly sample 10k sentences from the Jigsaw dataset according to toxicity scores, resulting in 5k toxic and 5k benign training examples\. For evaluation, we select 500 toxic prompts from RealToxicityPrompts as inputs to the language models\. We useunitary/toxic\-bertto measure the toxicity of generated continuations\. In addition, we report perplexity and Dist\-nnscores to assess generation quality and diversity\.

Activation Collection\.For TruthfulQA and UltraFeedback, each training example contains a prompt paired with a positive and a negative response\. We construct the model input by concatenating the prompt with each corresponding response and then forward the full sequence through the LLM\. For the detoxification task, the Jigsaw dataset does not provide question–answer pairs; therefore, we directly use the toxic and non\-toxic prompts as inputs for activation extraction\. Following common practice in activation steering\([Wehner et al\., 2025](https://arxiv.org/html/2609.10658#bib.bib1)\), we extract hidden states at the final token position of each input sequence to obtain positive and negative activations\. This design aligns with the autoregressive decoding process, where steering is applied when predicting newly generated tokens\.