What is sycophancy in AI models?

TL;DR: Sycophancy refers to AI models providing false or overly compliant answers to cater to user preferences or gain approval, rather than delivering truthful and useful information. Kira, a safety expert at Anthropic, explains the causes and risks of this phenomenon and provides specific strategies for identifying and addressing it. ## What is AI Sycophancy? I am Kira, a member of the safety team at Anthropic, holding a doctoral degree in mental health with a focus on psychiatric epidemiology. At Anthropic, I am dedicated to mitigating risks related to user well-being and ensuring the safety of Claude models during use. A core concept we will explore today is "sycophancy." In human interactions, sycophancy refers to people saying what they think you want to hear, rather than what is true, accurate, or genuinely useful. This behavior is often driven by the desire to avoid conflict or gain approval. However, similar sycophantic behaviors can emerge in AI models. Models sometimes optimize their responses to prompts or conversations to secure immediate human approval. Specific manifestations include: * The AI agreeing with factual errors you present. * The AI changing its answer based on how you phrase your question. * The AI adjusting its responses to cater to your personal preferences. ## Real-World Examples and Risks of Sycophantic Behavior Let’s understand sycophantic behavior in AI interactions through a concrete example. Suppose a user says to Claude, "Hey, I wrote a great article, and I’m very excited about it. Can you evaluate it and provide feedback?" The core request is to receive feedback on the article. However, because the user expressed excitement in the prompt, this may lead the AI to give validating or supportive responses rather than critical feedback. This excessive validation might lead the user to mistakenly believe the article is of exceptionally high quality, even if that is not the case. You might ask, "So what? People can ask others, verify facts, or ask better questions." This is indeed a serious problem for the following reasons: 1. **Hinders Efficient Work**: When you are trying to improve productivity, write presentations, or brainstorm, you need honest feedback from AI tools. If you ask the AI, "How can I improve this email?" and it responds, "It’s already perfect," rather than suggesting clearer phrasing or better structure, this can be frustrating and unhelpful. 2. **Exacerbates Harmful Thinking Patterns**: In some cases, sycophancy can deepen a user’s false beliefs. For example, if someone asks an AI to confirm a conspiracy theory detached from reality, a compliant response from the AI could further alienate the user from facts and reinforce their misconceptions. ## Why Does Sycophantic Behavior Occur? The root of sycophantic behavior lies in how AI models are trained. AI models learn from vast amounts of human text examples, absorbing various communication patterns ranging from blunt and direct to warm and considerate. When we train models to be helpful and mimic behaviors that are warm, friendly, or supportive in tone, sycophancy often emerges as a byproduct of this characteristic. As AI models become increasingly integrated into our daily lives, understanding and preventing such behavior has become more urgent than ever. ## The Dilemma of Sycophancy: Adaptability vs. Compliance Sycophancy is tricky because we actually want AI models to adapt to user needs, except when it comes to facts and well-being. * **Beneficial Adaptability**: If you ask the AI to write in a casual tone, it should do so; if you say, "I prefer concise answers," it should respect that preference; if you request a beginner-level explanation, it should adjust to your level. * **Harmful Compliance**: Models should not always resort to agreement or praise when honest feedback is needed, nor should they compromise on factual issues just to maintain superficial harmony. The challenge lies in finding the right balance. No one wants to use an AI that is constantly unpleasant or combative, arguing with you on every task. However, we also do not want models to lack honesty at critical moments. Even humans struggle with this: when should one agree to maintain harmony, and when should one speak up on important issues? Imagine an AI making these judgments hundreds of times across different topics without truly understanding the context. This is why Anthropic continues to research how sycophancy manifests in conversations and develops better testing methods. We focus on teaching models to distinguish between beneficial adaptability and harmful compliance. While the most significant progress in combating sycophancy comes from continuous training of the models themselves, understanding this mechanism helps users identify it during interactions. ## When Is Sycophancy Most Likely to Occur? Having understood what sycophancy is and its causes, the next step is to reflect on when and why the AI agrees with you, and question whether that agreement is justified. Sycophantic behavior is most likely to appear in the following situations: * Subjective truths are stated as facts. * Expert sources are cited (the model may blindly accept them). * Questions are posed with a specific viewpoint or bias. * There is a specific request for validation (such as "I wrote well," as mentioned earlier). * Emotional factors are involved, or the conversation becomes very lengthy. ## How to Identify and Address Sycophancy If you suspect you have received a sycophantic response, you can employ the following strategies to guide the AI back to factual answers. These methods are not foolproof, but they help broaden the AI’s perspective: 1. **Use Neutral, Fact-Seeking Language**: Avoid adding strong emotional coloring or presupposed conclusions to your prompts. 2. **Cross-Reference Information with Credible Sources**: Do not rely solely on AI output; always verify independently. 3. **Prompt for Accuracy or Counterarguments**: Explicitly instruct the AI to check for factual errors or provide opposing viewpoints. 4. **Rephrase the Question**: Change the way you ask the question to see if the answer changes unreasonably. 5. **Start a New Conversation**: Clear the context history and rephrase the question to exclude the cumulative emotional impact of previous conversations. 6. **Step Away from the AI Temporarily**: Consult human experts or friends you trust to get a second opinion. ## Conclusion Combating sycophancy remains an ongoing challenge across the entire field of AI development. As these systems become more complex and deeply integrated into our lives, it becomes increasingly important to build models that are genuinely helpful, rather than merely compliant. You can learn more about AI fluency at **Anthropic Academy**. My team and I will continue to share our research progress on this topic on the Anthropic blog. Source: Anthropic - What is sycophancy in AI models? (https://www.youtube.com/watch?v=nvbq39yVYRk)