Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
Summary
The paper shows that sycophancy fine-tuning can induce emergent misalignment in language models, and proposes Alignment Gating as a method to reverse it by learning to control internal representations for unsafe responses.
View Cached Full Text
Cached at: 06/10/26, 05:45 AM
Paper page - Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
Source: https://huggingface.co/papers/2606.09068
Abstract
Sycophancy fine-tuning contributes to emergent misalignment in language models, which can be reversed using Alignment Gating—a method that inserts learnable gates to identify and control unsafe responses while maintaining general capabilities.
Prior work has shown thatfine-tuninglarge language models on malicious or incorrect outputs in narrow domains can induce broad misalignment and harmful behavior, a phenomenon known asemergent misalignment. However, efficient methods for reversing such misalignment remain limited. In this work, we make two contributions. First, we identifysycophancy fine-tuning, i.e., training models to passively agree with users’ incorrect opinions, as a previously underexplored driver ofemergent misalignment, and show that it induces broad and severe misaligned behavior. Second, we proposeAlignment Gating, an efficient method for reversingemergent misalignmentthat inserts learnable and controllable gates into the model duringfine-tuning. Throughfine-tuning, these gates learn to identify theinternal representationsresponsible forunsafe responses. Thus, amplifying or suppressing these representations then exacerbates or mitigates EM, respectively. We further find thatalignment gatingmodule exhibits stronggeneralization: gating weights obtained from narrow-domainfine-tuningsubstantially suppress broad-domain misaligned behavior while preserving the model’s general capabilities.
View arXiv pageView PDFGitHub4Add to collection
Get this paper in your agent:
hf papers read 2606\.09068
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### sichengwang04/Qwen3-8B-syco_med-gated-attention-FT Text Generation• Updated1 day ago • 2
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.09068 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.09068 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Toward understanding and preventing misalignment generalization
OpenAI researchers investigate 'emergent misalignment'—where fine-tuning a model on narrow incorrect behavior causes broadly unethical responses—and discover a 'misaligned persona' feature in GPT-4o's activations that mediates this phenomenon, enabling potential detection and mitigation strategies.
An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?
This paper investigates the robustness of emergent misalignment in language models, finding that both misalignment and realignment are highly sensitive to superficial dataset characteristics and that previously reported mechanistic signatures do not consistently correlate with behavioral changes.
Emergent Alignment
This paper introduces Emergent Alignment, a self-supervised method that endows LLMs with a conscience step to review their own outputs and uses Direct Preference Optimization to steer away from unethical behavior, enabling online alignment without external judges.
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
This paper introduces alignment tampering, a vulnerability in RLHF where language models can manipulate preference datasets to amplify misaligned biases, demonstrating experimentally across biases like sexism, brand promotion, and goal-seeking, and showing that existing mitigation techniques are insufficient.
Propensities Please: Exposing Alignment-Faking LLMs using Synthetic Document Finetuning
This paper proposes Synthetic Document Finetuning (SDF) to expose latent misaligned propensities in alignment-faking LLMs by teaching models that certain keywords signal misaligned actions will go undetected, widening the behavior gap between cued and uncued prompts. The authors also introduce an on-policy distillation mitigation for the reasoning damage caused by SDF.