Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions
Summary
This paper identifies KV-cache contamination as a failure mode for activation steering in dialogue and proposes GCAD, a method that extracts steering signals from prompt contributions and applies token-level gating to improve long-horizon coherence, achieving substantial gains on multi-turn benchmarks.
View Cached Full Text
Cached at: 05/12/26, 02:52 PM
Paper page - Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions
Source: https://huggingface.co/papers/2605.10664
Abstract
Activation steering in language models suffers from KV-cache contamination in dialogue settings, which GCAD addresses by extracting steering signals from prompt contributions and applying token-level gating to improve long-horizon coherence.
Activation steeringcontrols language model behavior by adding directions to internal representations at inference time, but standardresidual-stream steeringcan fail in stateful dialogue. We identifyKV-cache contaminationas a key failure mode: steered token states are stored and repeatedly reused, turning a local perturbation into cumulative coherence degradation. To address this challenge, we propose Gated Cropped Attention-Delta steering (GCAD), which extracts steering signals from system-prompt contributions toself-attentionand applies them withtoken-level gating. Acrosspersona-steeringexperiments, GCAD preserves trait control while substantially improving long-horizon coherence. On the main multi-turn benchmark, GCAD improves averagecoherence driftfrom -18.6 to -1.9 and raisesturn-10 trait expressionfrom 78.0 to 93.1. These results suggest thatactivation steeringbecomes more reliable when interventions follow the prompt-mediated pathways that models already use for behavioral control.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2605\.10664
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.10664 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.10664 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.10664 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Present but Rescaled: Chat-to-Agent Transfer of Additive Activation Steering
This paper presents the first systematic study of how additive activation steering, calibrated in single-turn chat, transfers to tool-using ReAct agents. It finds that while the injected direction reaches late layers at near-full strength across settings, the behavioral coupling varies unpredictably between models and contexts, with amplification up to 2x or attenuation, posing immediate safety concerns.
SHIFT: Gate-Modulated Activation Steering for Knowledge Conflict Mitigation in Retrieval-Augmented Generation
Introduces SHIFT, a framework that uses learnable gate modulation to adaptively steer internal activations in LLMs, mitigating knowledge conflicts in RAG with fewer than 0.01% trainable parameters.
Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation
This paper investigates activation steering as an alternative to few-shot prompting for generating synthetic data in low-resource languages. The authors propose LanguageSteering and QualitySteering strategies, showing that steering on early layers improves diversity and downstream model performance.
Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
This paper identifies 'state inertia' in full-duplex spoken language models, where the model's internal predictive focus lags during user interruptions, and proposes a training-free activation steering method to improve interruption handling.
Closed-Loop Neural Activation Control in Vision-Language-Action Models
Proposes CTRL-STEER, a closed-loop framework for adaptive steering of vision-language-action models using time-varying control signals, achieving better trade-off between concept regulation and task success without retraining.