When is Your LLM Steerable?
Summary
This paper introduces a method to predict activation steering effectiveness in language models from early decoding states using a Gradient Boosting Decision Trees (GBDT) classifier, enabling efficient steering strength optimization without full rollouts.
View Cached Full Text
Cached at: 06/15/26, 09:06 AM
Paper page - When is Your LLM Steerable?
Source: https://huggingface.co/papers/2606.11599
Abstract
Activation steering effectiveness can be predicted from early decoding states using a GBDT classifier, enabling efficient steering strength optimization with reduced computational cost.
Activation steeringoffers a lightweight approach to controllanguage models’ behavior atinference time, but whether it succeeds or fails heavily depends on the prompt, concept, model, and steering configuration. Finding the regime and boundaries of successful steering typically requires expensivegrid searchesand post-hoc evaluation of fullautoregressive rollouts. In this work, we investigate whether steerability can be predicted from the model’s internal states at the beginning of the generation process, e.g., after generating the first few tokens, and how to leverage such a predictor to improve steering success rate. To this end, we first introduce ASTEER, a testbed including 1.4M steered generations, spanning 150 concepts with each steering success/failure labeled. Leveraging this testbed, we analyze the model’searly decoding dynamicsby extracting features that comparehidden statesbefore and after steering across layers and initial decoding steps. These features help us understand how steering’s effects propagate along layers and token positions, which provide key information forsteerability prediction. We then train aGradient Boosting Decision Trees(GBDT) classifier on these features to predict whether an intervention will under-steer, succeed, or over-steer without requiring full rollout. Our predictor achieves around 0.7 macro-F1 score on unseen concepts, demonstrating that earlyhidden statesencode substantial, structured information about eventual steering efficacy. We further leverage this steerability predictor as guidance forsteering strength searching, achieving near-optimal performance with a small fraction of decoding cost.
View arXiv pageView PDFGitHub3Add to collection
Get this paper in your agent:
hf papers read 2606\.11599
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.11599 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.11599 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.11599 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
When is Your LLM Steerable?
This paper investigates when activation steering succeeds or fails for LLMs by analyzing early decoding dynamics. The authors introduce ASTEER, a large testbed of steered generations, and train a GBDT classifier to predict steering outcomes from early hidden states, enabling efficient steering strength search.
Multi-Attribute Steering of Language Models via Targeted Intervention
MAT-Steer introduces a novel inference-time intervention framework for steering LLMs across multiple conflicting attributes by learning sparse, orthogonal steering vectors that selectively target tokens relevant to each attribute, achieving gains in QA tasks and generative tasks over prior methods.
Deployable Per-Instance Multi-Layer Activation Steering for Large Language Models
This paper introduces a deployable per-instance, multi-layer activation steering technique for large language models, showing that optimal layer selection varies per input and can be predicted from the prompt embedding without gold labels at inference.
SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
SALSA introduces a lightweight adaptation method for speech-aware LLMs that learns layer-wise steering vectors via supervised objective, achieving significant improvements (up to 46.8% relative) on out-of-domain speech benchmarks, and shows that steering the encoder layers is more effective than modifying the LLM backbone.
Steered LLM Activations are Non-Surjective
This paper proves that activation steering in LLMs produces internal states that cannot be replicated by any textual prompt, establishing a formal separation between white-box steerability and black-box prompting.