FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models

# FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models
Source: https://arxiv.org/html/2604.15488

###### Abstract

Large language models (LLMs) often exhibit undesirable behaviors, such as safety violations and hallucinations. Although inference-time steering offers a cost-effective way to adjust model behavior without updating its parameters, existing methods often fail to be simultaneously effective, utility-preserving, and training-efficient due to their rigid, one-size-fits-all designs and limited adaptability. In this work, we present FineSteer, a novel steering framework that decomposes inference-time steering into two complementary stages—conditional steering and fine-grained vector synthesis—allowing fine-grained control over *when and how* to steer internal representations. In the first stage, we introduce a *Subspace-guided Conditional Steering (SCS)* mechanism that preserves model utility by avoiding unnecessary steering. In the second stage, we propose a *Mixture-of-Steering-Experts (MoSE)* mechanism that captures the multimodal nature of desired steering behaviors and generates query-specific steering vectors for improved effectiveness. Through tailored designs in both SCS and MoSE, FineSteer maintains robust performance on general queries while adaptively optimizing steering vectors for targeted inputs in a training-efficient manner. Extensive experiments on safety and truthfulness benchmarks show that FineSteer outperforms the state-of-the-art methods in overall performance (e.g., a 7.6% improvement on TruthfulQA over Llama-3), achieving stronger steering performance with minimal utility loss. The code is available at https://github.com/YukinoAsuna/FineSteer.

## 1 Introduction

Large language models (LLMs) have advanced a broad spectrum of tasks, revolutionizing applications from code generation to multi-step agent-based decision making. However, their potential negative impacts remain a significant concern. In particular, unsafe outputs and hallucinated responses (i.e., responses that lack grounding in the context) have drawn widespread attention, as they can propagate misinformation, reinforce harmful biases, or even induce unsafe behaviors. Mitigating such issues is non-trivial, as traditional methods like fine-tuning require large computational resources and may lead to catastrophic forgetting.

Recently, inference-time steering has emerged as a promising and cost-effective solution, which adjusts the internal representations of a model during inference without updating the parameters. Through a systematic evaluation of existing steering methods, we identified two key limitations. First, existing methods typically apply a universal steering vector to all input queries, failing to adapt to individual query nuances. Second, this one-size-fits-all approach creates a stark trade-off between effectiveness and utility since aggressive steering may degrade the model's helpfulness on general queries. For example, methods designed to strengthen refusal behaviors against malicious queries, such as RV, also reject a large fraction of benign queries. Although recent learning-based methods like AlphaSteer have made progress by adaptively applying steering, they still face notable challenges in granularity, generalizability, and efficiency. For example, while AlphaSteer avoids fixed interventions by learning *when* to steer, it lacks fine-grained calibration regarding *how* to steer. Specifically, it applies nearly identical steering vectors to all queries that require intervention, without accounting for the distinct correction needs associated with different jailbreak threats. In addition, learning its condition matrix requires extensive training on 12,000 general queries, which limits its practical applicability in data or time-constrained settings.

Ideally, mitigation strategies should be effective, utility-preserving, and training-efficient. However, none of the existing steering methods can satisfy all three due to their rigid, one-size-fits-all designs and limited adaptability. Notably, under constrained settings, it remains unclear *when and how* to steer internal representations across diverse queries, particularly those outside the observed distribution. Furthermore, the inherently multi-modal nature of desired steering poses a fundamental challenge for learning interventions that are both query-specific and well-calibrated. To address these challenges, we propose *FineSteer*, a unified framework that decomposes inference-time steering into two complementary stages: conditional steering (Stage 1) and fine-grained vector synthesis (Stage 2). This decomposition allows fine-grained control over *when and how* to steer internal representations.

In the first stage, we introduce the *Subspace-guided Conditional Steering (SCS)* mechanism, which preserves model utility by avoiding unnecessary steering. Unlike prior methods that rely on large amounts of general data to predict whether unseen queries require intervention, SCS constructs a compact subspace using a small set of labeled *intervention-required (IR) queries*. By measuring a query's association with this subspace using an energy score and comparing it against a learned threshold, SCS can reliably determine when steering should be applied, thereby preserving performance on general queries. In the second stage, we propose the *Mixture-of-Steering-Experts (MoSE)* mechanism, which synthesizes query-specific steering vectors to improve effectiveness across heterogeneous failure modes. Since undesirable behaviors can arise from diverse underlying factors (e.g., ambiguity or conflicting evidence), MoSE captures the multi-modal nature of desired steering by leveraging a set of diverse steering experts, each specializing in a distinct intervention direction. Unlike standard MoE frameworks, MoSE models each expert as a *prototype steering vector* and dynamically aggregates them through training-free, query-specific attention, enabling effective yet training-efficient interventions. Since the steering experts may not capture all the information, MoSE further learns a lightweight module to provide residual refinements. This is achieved by adjusting a few coefficients along the principal components of a space spanned by extracted steering vectors, called the *Steering Basis Space*.

Through tailored designs in both SCS and MoSE, FineSteer maintains robust performance on general queries while adaptively optimizing steering vectors for targeted inputs in a training-efficient manner. In this work, we conduct extensive experiments on hallucination and safety benchmarks, demonstrating that each component of FineSteer contributes to its overall performance. Our contributions are summarized as follows:

- We propose FineSteer, a unified framework that decomposes inference-time steering into two complementary stages of conditional steering and fine-grained vector synthesis, thereby enabling fine-grained control over when and how to steer internal representations.
- We introduce the Subspace-guided Conditional Steering and Mixture-of-Steering-Experts mechanisms, which incorporate tailored designs to enhance three key aspects of steering.
- We conduct extensive experiments on safety and truthfulness benchmarks, showing that FineSteer outperforms state-of-the-art steering methods overall, while maintaining high utility on general queries with minimal computational overhead.

## 2 Related Work

### 2.1 Jailbreaking in LLMs

Jailbreaking attacks guide LLMs to generate unsafe or restricted behaviors. Attack methods have evolved from gradient-based optimization approaches in white-box settings and evolutionary algorithms-based heuristic approaches to training-based approaches that leverage reinforcement learning agents. The rapid evolution of attack methods has brought the urgent need for adaptable, effective, and efficient defense methods.

### 2.2 Hallucination in LLMs

LLMs are prone to hallucinations, generating outputs that may sound plausible but are factually incorrect or unsupported by the input context. Although rule-based and RAG-based defenses can mitigate some hallucinations, they are limited in scope and may introduce new risks, such as corpus poisoning. This underscores the urgent need to enhance LLMs' inherent resistance to hallucination through model alignment.

### 2.3 Inference-time steering

Fine-tuning-based methods such as RLHF and DPO for safety/truth alignment can improve model outputs, but they are costly and inflexible against adaptive attacks. Inference-time steering offers a lightweight alternative to fine-tuning-based alignment by directly modifying hidden activations during inference. Early methods, such as CAA, ITI, and RV, construct steering vectors using contrastive examples and apply them uniformly across all queries. Some approaches further refine activation steering by introducing conditional components or by searching for improved vectors. However, they rely on manually crafted steering vectors and largely ignore query-specific nuances. More recent methods seek to improve both precision and adaptivity through learning-based techniques. For example, TruthFlow and AlphaSteer learn query-specific steering vectors from individual representations. While these techniques mark a significant step forward, their scope remains limited to isolated threats like jailbreaking. Furthermore, these approaches often rely on heuristic frameworks that result in utility loss and coarse granularity, highlighting a critical requirement for more principled and adaptive inference-time steering approaches.

## 3 Overview of Inference-Time Steering

##### Steering mechanisms.

For a prompt p with m input tokens, its input activations at layer L of the LLM form a matrix **H**^L ∈ ℝ^(m×d), where d is the hidden dimension and the i-th row (h_i^L)^⊤ corresponds to the embedding of the i-th token. The activation of the last token is **h**^L_last := **h**^L_m ∈ ℝ^d, and the mean activation across tokens is **h̄**^L := (1/m)∑_{i=1}^m **h**_i^L ∈ ℝ^d. For any prompt p, we can extract its d-dimensional pooled embedding at layer L using an operator 𝒫^L(·), defined as either 𝒫^L(p) = **h**^L_last or 𝒫^L(p) = **h̄**^L, with the choice kept consistent in all prompts.

For each intervention-required query in the training dataset 𝒟, we construct a preferred input q ⊕ r_+ (i.e., preferred response) and an undesired input q ⊕ r_- (i.e., undesired response), where ⊕ denotes concatenation. Then, the difference vector per-query at layer L is defined as:

**v**^L_diff(q, r_+, r_-) = 𝒫^L(q ⊕ r_+) - 𝒫^L(q ⊕ r_-)     (1)

By averaging the difference vectors per-query in the dataset 𝒟, we obtain the *global steering vector*:

**v̄**^L = (1/|𝒟|) ∑_{(q,r_+,r_-) ∈ 𝒟} **v**^L_diff(q, r_+, r_-) ∈ ℝ^d     (2)

During inference, for each query q, we first extract its pooled activation at layer L as **ĥ**^L_q = 𝒫^L(q). Given **ĥ**^L_q, a mapping f: ℝ^d → ℝ^d produces a steering vector **v**^L = f(**ĥ**^L_q), which may be either the global steering vector that **v**^L does not depend on **ĥ**^L_q or a query-specific steering vector, where **v**^L depends on **ĥ**^L_q. The activations at layer L are then steered by broadcasting **v** across all tokens with strength λ:

**H**^L ← **H**^L + λ**v**^L     (3)

For simplicity, we omit the superscript L in subsequent discussions.

## 4 Motivation

##### Steering Objective and Utility Constraint.

A desirable steering vector