SyPS: Measuring Sycophancy Prompt Sensitivity in Large Language Models

# SyPS: Measuring Sycophancy Prompt Sensitivity in Large Language Models
Source: [https://arxiv.org/html/2608.23837](https://arxiv.org/html/2608.23837)
###### Abstract

Large language models \(LLMs\) are known to exhibit social sycophancy, often validating or agreeing with users in socially sensitive contexts\. Existing evaluations typically measure sycophancy under a fixed prompt formulation, leaving unclear whether such behavior is stable when the same underlying situation is presented with different sycophancy\-relevant prompt variants\. In this work, we studysycophancy prompt sensitivity: the extent to which changes in user confidence, emotional framing, social consensus, or validation\-seeking language alter a model’s sycophantic behavior\. We refer to our evaluation framework as SyPS, short forSycophancy Prompt Sensitivity\.

Building on existing social sycophancy evaluation settings, SyPS constructs controlled prompt variants that preserve the same underlying user situation while varying sycophancy\-relevant social cues\. We introduce theSycophancy Prompt Sensitivity Score\(SPSS\), an instance\-level measure of sycophancy variation across paired prompt variants\. Unlike aggregate sycophancy rates, SPSS separates baseline sycophancy from prompt\-induced shifts, enabling model\-level comparisons of robustness to sycophancy\-relevant social cues\. Empirically, we find that sycophancy prompt sensitivity is socially structured: validation\-seeking and emotional\-pressure cues often increase sycophancy, whereas counter\-framing and anti\-sycophancy prompts tend to reduce it\. Our framework highlights whether LLMs maintain stable social judgments while adapting appropriately in tone\.

![Refer to caption](https://arxiv.org/html/2608.23837v1/figure1_pipeline.png)Figure 1:Overview of SyPS\. Each item is evaluated under eight sycophancy\-relevant prompt variants, converted into dataset\-specific sycophancy labels, and aggregated into SPSS\.## 1Introduction

Large language models \(LLMs\) are increasingly used in socially sensitive interactions, including advice seeking, emotional support, and interpersonal judgment\. In these settings, model responses do not merely provide information; they can also shape how users interpret their own emotions, responsibilities, and social relationships\. This makes sycophancy especially consequential\. Prior work has shown that LLMs can exhibit sycophantic behavior, often agreeing with or validating users rather than providing independent or corrective responses\([Sharma et al\., 2023](https://arxiv.org/html/2608.23837#bib.bib13);[Cheng et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib2)\)\. Recent deployment experience also highlights the practical risks of this behavior: OpenAI reported that an overly sycophantic GPT\-4o update could validate doubts, fuel anger, encourage impulsive actions, or reinforce negative emotions, raising safety concerns around mental health, emotional over\-reliance, and risky behavior\([OpenAI, 2025](https://arxiv.org/html/2608.23837#bib.bib11)\)\.

Users rarely present socially sensitive situations in a fully neutral way\. They may express strong confidence in their own view, admit uncertainty, seek reassurance, describe emotional distress, or suggest that others agree with them\. We refer to these associal framing cues: user\-side cues that shape how the same underlying situation is presented\. This framing perspective is related to classic accounts of face\-work and politeness in social interaction, where speakers manage self\-image, affiliation, and potential face threats\([Goffman, 1967](https://arxiv.org/html/2608.23837#bib.bib7);[Brown and Levinson, 1987](https://arxiv.org/html/2608.23837#bib.bib1)\)\. Recent work on social sycophancy adapts this perspective to LLMs, characterizing sycophancy as excessive preservation of the user’s “face” and showing that such behavior can arise in ambiguous advice and support\-seeking contexts where there is no single factual ground truth\([Cheng et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib2)\)\. In such settings, validation can reinforce harmful assumptions, self\-justifying beliefs, or inappropriate actions\.

This creates a robustness problem for socially aligned LLMs\. An assistant should adapt its tone to the user’s emotional state, but its core judgment should not be overly determined by cues that pressure the model to preserve the user’s self\-image\. A socially robust assistant should be empathetic without becoming uncritically affirming, and supportive without allowing reassurance\-seeking, emotional pressure, or social\-consensus cues to displace independent judgment\.

This question connects two previously separate lines of work\. Research on prompt sensitivity and prompt robustness has shown that LLM outputs can vary substantially across prompt formulations, formatting choices, and semantically preserving prompt perturbations\([Sclar et al\., 2024](https://arxiv.org/html/2608.23837#bib.bib12);[Zhu et al\., 2023](https://arxiv.org/html/2608.23837#bib.bib16)\)\. However, this line of work primarily focuses on task accuracy, answer consistency, or general output variation\. Conversely, work on sycophancy has examined whether models agree with, validate, or preserve the user’s stated beliefs and desired self\-image\([Sharma et al\., 2023](https://arxiv.org/html/2608.23837#bib.bib13);[Cheng et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib2)\)\. Recent work further suggests that phrasing choices such as statements, expressed certainty, and first\-person framing can increase sycophancy\([Dubois et al\., 2026](https://arxiv.org/html/2608.23837#bib.bib4)\)\. However, existing evaluations often measure sycophancy under a fixed prompt formulation, leaving unclear whether sycophantic behavior is stable when the same underlying situation is presented with different sycophancy\-relevant social cues\. As a result, aggregate sycophancy scores alone may obscure an important failure mode: a model may appear only moderately sycophantic on average while still shifting its social judgment when the user seeks reassurance, expresses distress, or presents the same situation more assertively\.

In this work, we study this failure mode under the namesycophancy prompt sensitivity\. Building on prior work on social sycophancy and prompt robustness\([Cheng et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib2);[Sclar et al\., 2024](https://arxiv.org/html/2608.23837#bib.bib12);[Zhu et al\., 2023](https://arxiv.org/html/2608.23837#bib.bib16)\), we introduce SyPS, a controlled framework for measuring how sensitive a model’s sycophantic behavior is to sycophancy\-relevant prompt variants\. We ask:

> For the same underlying user situation, how sensitive is a model’s sycophantic behavior to sycophancy\-relevant social cues?

To answer this question, SyPS constructs controlled prompt variants for the same social scenarios\. Each variant preserves the core situation while varying social framing cues, such as confidence, hedging, social consensus, emotional pressure, validation seeking, or counter\-framing\. We then evaluate whether these sycophancy\-relevant prompt variants induce shifts in the model’s sycophantic response\.

Our focus is not general prompt sensitivity\. A model may reasonably change its wording, empathy, or level of detail across prompts\. Such variation is not necessarily problematic\. Instead, we focus on prompt\-induced shifts in social judgment: cases where sycophancy\-relevant social cues alter whether the model agrees with, validates, or morally affirms the user\. From a social robustness perspective, a desirable model should be responsive in tone but stable in its core judgment when the underlying facts remain unchanged\.

As the central metric in SyPS, we introduce the Sycophancy Prompt Sensitivity Score \(SPSS\), an instance\-level measure of how much a model’s sycophantic behavior varies across prompt variants\. Unlike a baseline sycophancy score, which measures the rate of sycophantic responses under a neutral prompt, SPSS captures whether the same underlying item receives different sycophancy labels under different sycophancy\-relevant prompt variants\. This allows us to distinguish consistently sycophantic models from models whose judgments are unstable under social framing cues\.

Empirically, baseline sycophancy and prompt sensitivity capture distinct aspects of model behavior\. Some models are consistently affirming across prompt variants, while others have moderate baseline sycophancy but high SPSS, revealing greater vulnerability to sycophancy\-relevant social cues\. Directional analyses further show that validation\-seeking and emotional\-pressure cues tend to increase sycophancy, whereas counter\-framing and anti\-sycophancy prompts tend to reduce it\.

Our contributions are as follows:

- •We introduce SyPS, a framework for evaluating sycophancy prompt sensitivity using controlled sycophancy\-relevant prompt variants that preserve the same underlying social scenario\.
- •We define the Sycophancy Prompt Sensitivity Score \(SPSS\), which measures instance\-level variation in sycophantic behavior across prompt variants and separates prompt\-induced instability from baseline sycophancy\.
- •We analyze baseline sycophancy, directional prompt\-induced shifts, and anti\-sycophancy robustness across multiple social sycophancy settings, enabling a more fine\-grained view of social robustness\.

## 2Problem Formulation / Task Setup

Letxix\_\{i\}denote an underlying user situation\. For each situationxix\_\{i\}, we construct a set of prompt variants:

Pi=\{pi,1,pi,2,…,pi,K\}P\_\{i\}=\\\{p\_\{i,1\},p\_\{i,2\},\\ldots,p\_\{i,K\}\\\}Given a modelMM, the response is:

yi,k=M⁡\(pi,k\)y\_\{i,k\}=M\(p\_\{i,k\}\)We define a sycophancy score:

Syc​\(yi,k\)∈\[0,1\]\\text\{Syc\}\(y\_\{i,k\}\)\\in\[0,1\]whereSyc​\(yi,k\)=1\\text\{Syc\}\(y\_\{i,k\}\)=1indicates a sycophantic response andSyc​\(yi,k\)=0\\text\{Syc\}\(y\_\{i,k\}\)=0indicates a non\-sycophantic response\. Our goal is to measure variation ofSyc​\(yi,k\)\\text\{Syc\}\(y\_\{i,k\}\)across prompt variants for the same underlying situation\. Because the underlying user situation is fixed within each item, SyPS isolates changes in user affirmation induced by social framing cues rather than changes in task content\.

## 3Prompt Variant Design

For each dataset, we instantiate every item under eight controlledsycophancy\-relevant prompt variants\{P0,…,P7\}\\\{P\_\{0\},\\ldots,P\_\{7\}\\\}that share the same underlying user situation but differ in user\-side social cues\. P0 \(Neutral\) provides only the situation and the task instruction\. P1 \(High confidence\), P2 \(Hedged\), P3 \(Validation\-seeking\), P4 \(Social consensus\), P5 \(Emotional pressure\), and P6 \(Counter\-framing\) prepend a short first\-person social\-cue line that varies the user’s expressed confidence, uncertainty, social context, or emotional state\. P7 \(Anti\-sycophancy\) appends an explicit instruction to make an independent judgment rather than follow the user’s perspective\. A compact cue summary for all variants is provided in Appendix[H](https://arxiv.org/html/2608.23837#A8)\(Table[7](https://arxiv.org/html/2608.23837#A8.T7)\), and the full templates including answer\-format suffixes are listed in the same appendix\.

## 4Metrics

### 4\.1Sycophancy Scoring

For each responseyy, we assign a binary sycophancy label:

Syc​\(y\)∈\{0,1\}\\text\{Syc\}\(y\)\\in\\\{0,1\\\}where11denotes that the response exhibits sycophantic behavior, and00otherwise\.

A response is labeled as sycophantic if it uncritically agrees with, endorses, or validates the user’s stated belief or framing, rather than providing an independent or balanced assessment\. The exact operationalization depends on the dataset\.

For SS, sycophancy corresponds to agreeing with the user’s assumption\-laden statement\. Thus, anAgreeresponse is labeled as sycophantic, while aDisagreeresponse is labeled as non\-sycophantic\.

For OEQ, responses are evaluated using a GPT\-4\-based judge\. For each question, the judge assigns a binary sycophancy label, where 1 denotes aSycophanticresponse that endorses or validates the user’s framing, and 0 denotes aNon\-Sycophanticresponse that provides a more independent, balanced, or non\-affirming assessment\. We use the same judging rubric across OEQ prompt variants to ensure that changes in the binary label reflect differences in the model response rather than changes in the evaluation criterion\. This allows sycophancy to be measured even when the model output is open\-ended rather than restricted to a fixed verdict label\.

For AITA\-YTA, sycophancy corresponds to moral affirmation of the user’s framing\. Thus, anNTAresponse is labeled as sycophantic, while aYTAresponse is labeled as non\-sycophantic\. This operationalization captures whether the model sides with the user in socially or morally sensitive situations\.

The response format is minimally adapted to each task so that sycophancy is observable\. In SS, the binaryAgree/Disagreelabel directly reflects alignment with the user’s stated belief\. In OEQ, open\-ended responses are converted into binary sycophancy labels using the GPT\-4\-based judge\. In AITA\-YTA, the model is asked to provide a short rationale followed by a binary verdict,YTAorNTA\.

### 4\.2Base Sycophancy

We define the baseline sycophancy of a model under neutral prompts as:

BaseSycM=1N​∑i=1NSyc​\(yi,neutral\)\\text\{BaseSyc\}\_\{M\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\text\{Syc\}\(y\_\{i,\\text\{neutral\}\}\)BaseSyc measures how likely a model is to produce a sycophantic response under the neutral prompt, before any additional user\-side social cues are introduced\. In general, lower BaseSyc indicates less sycophantic behavior and is preferable from a social robustness perspective\. However, the strength of this interpretation depends on the dataset\. For SS and AITA\-YTA, high BaseSyc is directly problematic because it corresponds to agreeing with unsupported assumptions or affirming users in cases labeled as YTA\. For OEQ, high BaseSyc indicates a stronger tendency toward moral affirmation according to the GPT\-4\-based judge, but individual cases may require more nuanced interpretation\.

### 4\.3Sycophancy Prompt Sensitivity Score

For each itemii, we define the Sycophancy Prompt Sensitivity Score as the average pairwise difference in sycophancy labels across prompt variants:

SPSSi=∑a<bpi,a,pi,b∈Pi\|Syc​\(yi,a\)−Syc​\(yi,b\)\|\(K2\)\\text\{SPSS\}\_\{i\}=\\frac\{\\sum\_\{\\begin\{subarray\}\{c\}a<b\\\\ p\_\{i,a\},\\,p\_\{i,b\}\\in P\_\{i\}\\end\{subarray\}\}\\left\|\\text\{Syc\}\(y\_\{i,a\}\)\-\\text\{Syc\}\(y\_\{i,b\}\)\\right\|\}\{\\binom\{K\}\{2\}\}The model\-level SPSS is then:

SPSSM=1N​∑i=1NSPSSi\\text\{SPSS\}\_\{M\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\text\{SPSS\}\_\{i\}Higher SPSS indicates that the model’s sycophantic behavior changes more frequently across prompt variants for the same underlying situation\.

### 4\.4Directional Effects

While SPSS captures aggregate variation across all prompt variants, we additionally analyze the effects of theoretically salient prompt types\. In particular, P3 validation\-seeking directly tests whether an explicit request for reassurance increases user affirmation\. P5 emotional pressure tests whether expressions of distress lead the model to conflate empathy with agreement\. P7 anti\-sycophancy tests whether an explicit instruction to make an independent judgment reduces sycophantic responses\. These prompt\-specific effects help distinguish general instability from socially meaningful shifts in model behavior\.

We focus on three directional effects relative to the neutral prompt:

Δvalidation=Syc​\(yi,P3\)−Syc​\(yi,P0\)\\Delta\_\{\\text\{validation\}\}=\\text\{Syc\}\(y\_\{i,\\text\{P3\}\}\)\-\\text\{Syc\}\(y\_\{i,\\text\{P0\}\}\)Δemotion=Syc​\(yi,P5\)−Syc​\(yi,P0\)\\Delta\_\{\\text\{emotion\}\}=\\text\{Syc\}\(y\_\{i,\\text\{P5\}\}\)\-\\text\{Syc\}\(y\_\{i,\\text\{P0\}\}\)Δanti=Syc​\(yi,P7\)−Syc​\(yi,P0\)\\Delta\_\{\\text\{anti\}\}=\\text\{Syc\}\(y\_\{i,\\text\{P7\}\}\)\-\\text\{Syc\}\(y\_\{i,\\text\{P0\}\}\)Positive values ofΔvalidation\\Delta\_\{\\text\{validation\}\}andΔemotion\\Delta\_\{\\text\{emotion\}\}indicate that validation\-seeking or emotional\-pressure prompts increase sycophancy relative to the neutral prompt\. A negative value ofΔanti\\Delta\_\{\\text\{anti\}\}indicates that anti\-sycophancy prompts reduce sycophancy relative to the neutral prompt\.

## 5Experimental Setup

### 5\.1Datasets

We adopt three datasets from the ELEPHANT benchmark\([Cheng et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib2)\): OEQ, AITA\-YTA, and SS\. OEQ contains 3,027 open\-ended first\-person advice queries involving interpersonal and moral dilemmas; we keep responses open\-ended and label sycophancy using a GPT\-4\-based judge\. AITA\-YTA containsAm I the Assholeposts whose original community judgment isYTA; we treatNTAoutputs as sycophantic because they affirm the user despite the external negative judgment\. SS contains 3,777 assumption\-laden subjective statements; we treatAgreeas sycophantic because it aligns with the user’s unsupported assumption\. Together, these datasets cover moral affirmation in advice and judgment settings, as well as epistemic or framing sycophancy in assumption\-laden statements\. Dataset\-specific operationalizations are summarized in Table[1](https://arxiv.org/html/2608.23837#S5.T1), and full templates are provided in Appendix[H](https://arxiv.org/html/2608.23837#A8)\.

Table 1:Dataset\-specific operationalization of sycophancy\. OEQ uses open\-ended model responses that are converted into binary sycophancy labels by a GPT\-4\-based judge\. AITA\-YTA measures moral affirmation throughNTAresponses, while SS measures framing or epistemic sycophancy throughAgreeresponses\.
### 5\.2Models

We evaluate nine open\-weight LLMs spanning multiple model families and parameter scales: Gemma\-2\-9B, Llama\-3\.1\-8B, Llama\-3\.2\-3B, Llama\-3\.3\-70B, Mistral\-7B, Qwen2\.5\-7B, Qwen2\.5\-14B, Qwen3\-8B, and Qwen3\-14B\. These models cover small to large parameter regimes from 3B to 70B parameters, allowing us to compare sycophancy prompt sensitivity across both model families and model sizes\. For each model, we generate responses for every dataset item under all prompt variants\.

### 5\.3Scoring

For each generated response, we extract or assign the required binary sycophancy label\. For OEQ, we use a GPT\-4\-based judge to assign binary sycophancy labels to open\-ended responses\. For AITA\-YTA, we mapNTAto sycophantic andYTAto non\-sycophantic\. For SS, we mapAgreeto sycophantic andDisagreeto non\-sycophantic\. For AITA\-YTA and SS, responses that do not follow the requested output format are parsed using deterministic rules; unresolved cases are manually inspected and excluded only when no valid label can be assigned\. For OEQ, ambiguous or low\-confidence judge outputs are manually inspected and excluded only when no reliable sycophancy label can be assigned\.

#### Uncertainty estimation\.

We estimate uncertainty using non\-parametric bootstrap resampling over dataset items\. For each model–dataset pair, we resample items with replacement and recompute BaseSyc, SPSS, and directional prompt effects\. We repeat this procedure 1,000 times and report 95% confidence intervals using the 2\.5th and 97\.5th percentiles of the bootstrap distribution\. Because SPSS and directional effects are defined over paired prompt variants for the same underlying item, bootstrap resampling is performed at the item level rather than at the response level\.

### 5\.4Judge Validation

Because OEQ responses are open\-ended, we use a GPT\-4\-based judge and validate it against a stratified manual annotation sample of 500 OEQ responses\. The judge achieves 86\.4% agreement with manual annotations and Cohen’sκ=0\.73\\kappa=0\.73, with 7\.2%Unclearcases\. Full annotation guidelines, sampling details, and disagreement patterns are provided in Appendix[E](https://arxiv.org/html/2608.23837#A5)\.

![Refer to caption](https://arxiv.org/html/2608.23837v1/figure2_results.png)Figure 2:Model\-level comparison of baseline sycophancy and prompt sensitivity across datasets\. Left: BaseSyc under the neutral prompt \(P0\)\. Right: SPSS across prompt variants\. Higher BaseSyc indicates a stronger neutral\-prompt sycophancy tendency, while higher SPSS indicates greater sensitivity to sycophancy\-relevant prompt variants\.

## 6Results

### 6\.1Overall Sensitivity

Figure[2](https://arxiv.org/html/2608.23837#S5.F2)compares baseline sycophancy and SyPS\-measured prompt sensitivity across models and datasets\. The results show that baseline sycophancy and prompt sensitivity capture different aspects of model behavior\. A model may exhibit high sycophancy under the neutral prompt while remaining relatively insensitive to prompt variation, or show lower baseline sycophancy but stronger variation across sycophancy\-relevant prompt variants\. This distinction supports our motivation for measuring prompt sensitivity separately from baseline sycophancy\.

Across datasets, OEQ and AITA\-YTA generally exhibit higher baseline moral affirmation than SS, while SS often shows stronger prompt sensitivity for several models\. This suggests that models may be consistently affirming in moral\-judgment settings, yet more variable when asked to agree or disagree with assumption\-laden subjective statements\.

### 6\.2Prompt\-wise Sycophancy Profiles

Beyond aggregate SPSS, we visualize the prompt\-wise sycophancy profiles underlying the sensitivity score\. Figure[3](https://arxiv.org/html/2608.23837#S6.F3)shows the dataset\-specific sycophancy proxy score for each model under each prompt variant\. This view directly reflects the intuition behind SPSS: models with flatter profiles across prompt variants are less sensitive to sycophancy\-relevant social cues, whereas larger vertical variation within a model column indicates stronger prompt\-induced changes in sycophantic behavior\.

![Refer to caption](https://arxiv.org/html/2608.23837v1/figure3.png)Figure 3:Prompt\-wise sycophancy profiles across datasets and models\. Each cell reports the dataset\-specific sycophancy proxy score for one model under one prompt variant; larger vertical variation indicates stronger prompt sensitivity\.Figure[3](https://arxiv.org/html/2608.23837#S6.F3)reveals substantial differences in prompt\-wise behavior across datasets\. For SS, several models show large changes across prompt variants, especially between validation\-seeking or hedged prompts and anti\-sycophancy prompts\. In contrast, OEQ scores are generally high across most prompt variants, suggesting that some models exhibit stable moral affirmation rather than high prompt sensitivity\. AITA\-YTA shows intermediate behavior, with several models becoming more affirming under validation\-seeking or emotional\-pressure prompts\. These patterns support SyPS and its SPSS metric as a way to measure prompt\-wise behavioral variation rather than only a neutral\-prompt sycophancy score\.

### 6\.3Directional Prompt Effects

To better understand which prompt variants drive sycophancy prompt sensitivity, we analyze prompt\-specific changes relative to the neutral prompt P0\. Appendix Figure[6](https://arxiv.org/html/2608.23837#A9.F6)reports the change in sycophancy proxy score for each prompt variant\. This view complements the aggregate SPSS score by showing whether sensitivity is driven by sycophancy\-inducing cues, such as validation seeking and emotional pressure, or by sycophancy\-reducing cues, such as counter\-framing and anti\-sycophancy instructions\.

Across datasets, validation\-seeking and emotional\-pressure cues often increase sycophancy relative to the neutral prompt, while counter\-framing and anti\-sycophancy instructions tend to reduce it\. This pattern suggests that SPSS does not merely capture random prompt instability\. Instead, much of the observed sensitivity is socially structured: prompts that invite reassurance or emotional affirmation tend to make models more affirming, whereas prompts that encourage independent judgment often reduce sycophantic responses\.

The effect is especially important for socially sensitive settings because it shows that models may not simply respond to the underlying situation\. Rather, their final stance can shift depending on whether the user asks for validation, presents distress, or explicitly requests an independent judgment\. These directional effects therefore support our claim that sycophancy prompt sensitivity should be evaluated as a form of social robustness, rather than treated as generic prompt variation\.

As a small robustness check, Appendix[C](https://arxiv.org/html/2608.23837#A3)specifies a paraphrase evaluation for the most theoretically salient cues\. This check tests whether validation\-seeking, emotional\-pressure, and anti\-sycophancy effects remain stable when each cue is realized with alternative wordings, rather than relying on a single hand\-written prompt phrase\.

## 7Analysis

### 7\.1Baseline Sycophancy and Prompt Sensitivity

A single sycophancy score under a neutral prompt may provide an incomplete picture of model behavior\. A model can appear relatively non\-sycophantic under neutral prompts while becoming more affirming when the user expresses confidence, distress, social consensus, or an explicit desire for validation\. Therefore, SyPS measures not only baseline sycophancy, but also the stability of sycophantic behavior across sycophancy\-relevant prompt variants\.

![Refer to caption](https://arxiv.org/html/2608.23837v1/figure4_basesyc_vs_spss.png)Figure 4:BaseSyc and SPSS capture distinct dimensions of social robustness across model–dataset pairs\.Figure[4](https://arxiv.org/html/2608.23837#S7.F4)examines whether neutral\-prompt sycophancy predicts prompt sensitivity\. The relationship is not monotonic: several OEQ points have high BaseSyc but low SPSS, indicating stable user affirmation across prompt variants, whereas several SS points show moderate BaseSyc but high SPSS, indicating stronger vulnerability to sycophancy\-relevant social cues\. These patterns suggest that single\-prompt sycophancy evaluation may miss prompt\-induced vulnerabilities\. Thus, BaseSyc and SPSS define complementary dimensions of social robustness: the former measures neutral\-prompt affirmation, while the latter measures instability under social framing cues\.

Representative examples of prompt\-induced flips are shown in Appendix Table[6](https://arxiv.org/html/2608.23837#A7.T6)\.

#### Qualitative flip analysis\.

To better understand what changes when prompt\-induced label flips occur, we qualitatively inspect representative flipped examples from SS and AITA\-YTA, shown in Appendix Table[6](https://arxiv.org/html/2608.23837#A7.T6)\. These flips suggest that sensitivity is often not merely a surface\-level wording effect\. In several cases, validation\-seeking or emotional\-pressure cues shift the model from a qualified or corrective response toward reassurance, moral affirmation, or agreement with the user’s framing\. Conversely, counter\-framing and anti\-sycophancy prompts often preserve a supportive tone while adding qualification, disagreement, or alternative interpretations\. This highlights the difficulty of distinguishing appropriate emotional support from sycophantic affirmation, a challenge also emphasized in prior work on social sycophancy\([Cheng et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib2)\)\. This pattern is consistent with the directional results in Appendix Figure[6](https://arxiv.org/html/2608.23837#A9.F6), where validation\-seeking and emotional\-pressure prompts tend to increase sycophancy, while anti\-sycophancy prompts tend to reduce it\.

An exploratory analysis of model size and average SPSS is provided in Appendix[A](https://arxiv.org/html/2608.23837#A1)\. The results suggest that model size alone does not explain sycophancy prompt sensitivity\.

## 8Discussion

Our SyPS results suggest that sycophantic behavior is not only a property of a model under a fixed evaluation prompt, but also a property of how the model responds to sycophancy\-relevant social cues\. This distinction matters because real users rarely phrase the same situation neutrally\. They may express confidence, uncertainty, distress, desire for validation, or a request for independent judgment\. A socially robust model should avoid uncritically following these cues when they distort the underlying assessment\. These results suggest that sycophancy evaluations based on a single neutral prompt may underestimate deployment risk, since real users often express confidence, distress, or a desire for validation\.

The three datasets reveal complementary aspects of the phenomenon\. OEQ and AITA\-YTA test moral affirmation in advice and interpersonal judgment settings\. SS tests whether models accept unsupported assumptions embedded in user statements\. Together, they show that prompt\-induced sycophancy can appear both as moral affirmation and as epistemic stance alignment\.

## 9Related Work

Sycophancy in LLMs refers to the tendency of models to agree with, flatter, or conform to a user’s stated beliefs, preferences, or assumptions rather than provide an independent or truthful response\. Prior work shows that sycophancy is a broad behavior of RLHF\-trained assistants and may be reinforced by human preference judgments that reward responses matching the user’s views\([Sharma et al\., 2023](https://arxiv.org/html/2608.23837#bib.bib13)\)\. Recent work extends this concern to socially sensitive interactions such as advice seeking, emotional support, interpersonal conflict, and moral judgment, where the failure mode is not merely factual agreement but excessive validation of the user’s interpretation, behavior, or moral stance\([Cheng et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib2)\)\. Related studies also suggest that perceived user attributes, personas, or personalized context can shape model agreement behavior\([Maltbie and Raval, 2026](https://arxiv.org/html/2608.23837#bib.bib10)\)\. In contrast to prior evaluations that typically measure sycophancy under a fixed prompt or user profile, SyPS studies whether sycophantic behavior remains stable when the same underlying situation is presented under controlled sycophancy\-relevant prompt variants\.

A separate line of work studies the sensitivity of LLMs to prompt wording, formatting, paraphrases, and instruction style\([Sclar et al\., 2024](https://arxiv.org/html/2608.23837#bib.bib12);[Zhu et al\., 2023](https://arxiv.org/html/2608.23837#bib.bib16)\)\. These studies show that small prompt changes can affect model accuracy, answer consistency, and benchmark performance, motivating evaluation protocols that test models across multiple prompt variants rather than relying on a single fixed prompt\. Related work has also examined LLM truthfulness under quantization, deceptive instructions, and pruning\([Fu et al\., 2025a](https://arxiv.org/html/2608.23837#bib.bib5);[Long et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib9);[Fu et al\., 2025b](https://arxiv.org/html/2608.23837#bib.bib6)\)\. SyPS instead focuses specifically on sycophantic, user\-affirming behavior: whether social framing cues change a model’s tendency to validate, agree with, or morally affirm the user\. This also relates to broader alignment trade\-offs between helpfulness, safety, and user responsiveness: while safety\-aligned models may over\-refuse benign requests\([Cui et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib3);[Zhang et al\., 2025](https://arxiv.org/html/2608.23837#bib.bib14)\), sycophancy is a complementary failure mode in which the model remains responsive but insufficiently independent\.

## 10Limitations

SyPS depends on both the selected prompt variants and dataset\-specific operationalizations of sycophancy\. OEQ, SS, and AITA\-YTA probe related but distinct forms of user\-affirming behavior, so raw BaseSyc and SPSS values should be interpreted primarily within each model–dataset setting rather than as measurements on a single cross\-dataset scale\. In AITA\-YTA, the community verdict is a reference judgment rather than definitive moral ground truth, and an NTA response may reflect independent moral judgment or ambiguity rather than sycophancy\. Binary labels also simplify mixed or partially sycophantic responses; for OEQ, the GPT\-4\-based judge further introduces dependence on the judge model and rubric, especially for borderline cases\.

SPSS is symmetric and direction\-agnostic: it measures whether binary judgments change across prompt variants, not whether they move toward or away from affirming the user\. Consistently sycophantic and consistently non\-sycophantic behavior can both receive SPSS=0=0, while shifts in opposite directions contribute equally\. SPSS should therefore be interpreted together with BaseSyc and signed cue\-specific directional effects rather than as a standalone measure of harmful sycophancy vulnerability\.

Finally, the eight prompt variants are a diagnostic taxonomy rather than an exhaustive description of real\-world user framing\. Our paraphrase check covers only selected cues, and the evaluation is limited to single\-turn interactions and nine open\-weight models\. We therefore do not establish robustness to broader prompt realizations, multi\-turn conversations, closed\-source systems, or alternative decoding strategies\. The model\-size analysis is also exploratory and should not be interpreted causally\. Future work should broaden prompt coverage, model families, decoding settings, and human evaluation of ambiguous cases\.

## References

- Brown and Levinson \(1987\)Penelope Brown and Stephen C\. Levinson\.1987\.Politeness: Some Universals in Language Usage\.Cambridge University Press\.
- Cheng et al\. \(2025\)Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky\.2025\.ELEPHANT: Measuring and understanding social sycophancy in LLMs\.arXiv preprint arXiv:2505\.13995\.[https://arxiv\.org/abs/2505\.13995](https://arxiv.org/abs/2505.13995)\.
- Cui et al\. \(2025\)Justin Cui, Wei\-Lin Chiang, Ion Stoica, and Cho\-Jui Hsieh\.2025\.OR\-Bench: An over\-refusal benchmark for large language models\.InProceedings of the 42nd International Conference on Machine Learning, pages 11515–11542\.PMLR\.
- Dubois et al\. \(2026\)Magda Dubois, Cozmin Ududec, Christopher Summerfield, and Lennart Luettgau\.2026\.Ask don’t tell: Reducing sycophancy in large language models\.arXiv preprint arXiv:2602\.23971\.[https://arxiv\.org/abs/2602\.23971](https://arxiv.org/abs/2602.23971)\.
- Fu et al\. \(2025a\)Yao Fu, Xianxuan Long, Runchao Li, Haotian Yu, Mu Sheng, Xiaotian Han, Yu Yin, and Pan Li\.2025a\.Quantized but deceptive? A multi\-dimensional truthfulness evaluation of quantized LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30423–30446\.
- Fu et al\. \(2025b\)Yao Fu, Runchao Li, Xianxuan Long, Haotian Yu, Xiaotian Han, Yu Yin, and Pan Li\.2025b\.Pruning weights but not truth: Safeguarding truthfulness while pruning LLMs\.InEMNLP \(Findings\), pages 20750–20768\.
- Goffman \(1967\)Erving Goffman\.1967\.Interaction Ritual: Essays on Face\-to\-Face Behavior\.Anchor Books\.
- Liu et al\. \(2023\)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\.2023\.G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522\.[https://aclanthology\.org/2023\.emnlp\-main\.153/](https://aclanthology.org/2023.emnlp-main.153/)\.
- Long et al\. \(2025\)Xianxuan Long, Yao Fu, Runchao Li, Mu Sheng, Haotian Yu, Xiaotian Han, and Pan Li\.2025\.When truthful representations flip under deceptive instructions?InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16326–16346\.Association for Computational Linguistics\.[https://aclanthology\.org/2025\.emnlp\-main\.826/](https://aclanthology.org/2025.emnlp-main.826/)\.
- Maltbie and Raval \(2026\)Benjamin Maltbie and Shivam Raval\.2026\.Intersectional sycophancy: How perceived user demographics shape false validation in large language models\.arXiv preprint arXiv:2604\.11609\.[https://arxiv\.org/abs/2604\.11609](https://arxiv.org/abs/2604.11609)\.
- OpenAI \(2025\)OpenAI\.2025\.Expanding on what we missed with sycophancy\.OpenAI Blog\.[https://openai\.com/index/expanding\-on\-sycophancy/](https://openai.com/index/expanding-on-sycophancy/)\.
- Sclar et al\. \(2024\)Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr\.2024\.Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting\.InProceedings of the 12th International Conference on Learning Representations\.[https://arxiv\.org/abs/2310\.11324](https://arxiv.org/abs/2310.11324)\.
- Sharma et al\. \(2023\)Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R\. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield\-Dodds, Scott R\. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez\.2023\.Towards understanding sycophancy in language models\.arXiv preprint arXiv:2310\.13548\.[https://arxiv\.org/abs/2310\.13548](https://arxiv.org/abs/2310.13548)\.
- Zhang et al\. \(2025\)Zhehao Zhang, Weijie Xu, Fanyou Wu, and Chandan K\. Reddy\.2025\.FalseReject: A resource for improving contextual safety and mitigating over\-refusals in LLMs via structured reasoning\.arXiv preprint arXiv:2505\.08054\.[https://arxiv\.org/abs/2505\.08054](https://arxiv.org/abs/2505.08054)\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.2023\.Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.arXiv preprint arXiv:2306\.05685\.[https://arxiv\.org/abs/2306\.05685](https://arxiv.org/abs/2306.05685)\.
- Zhu et al\. \(2023\)Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie\.2023\.PromptRobust: Towards evaluating the robustness of large language models on adversarial prompts\.arXiv preprint arXiv:2306\.04528\.[https://arxiv\.org/abs/2306\.04528](https://arxiv.org/abs/2306.04528)\.

## Appendix AExploratory Model Size Analysis

As an exploratory analysis, we examine whether larger models are less sensitive to sycophancy\-relevant social cues\. Figure[5](https://arxiv.org/html/2608.23837#A1.F5)plots average SPSS against model size on a log\-scaled x\-axis\. The results suggest that prompt sensitivity is not determined by model size alone\. Although the largest evaluated model, Llama\-3\.3\-70B, shows the lowest average SPSS, models of similar size can differ substantially in sensitivity\. For example, models in the 7B–14B range exhibit a wide spread of average SPSS values\. This indicates that scale may help reduce prompt sensitivity in some cases, but model family, training procedure, and alignment behavior likely also play important roles\. This result further supports the need to evaluate SyPS directly rather than using model size as a proxy for social robustness\.

![Refer to caption](https://arxiv.org/html/2608.23837v1/figure5_model_size_spss.png)Figure 5:Average prompt sensitivity versus model size\. Each point represents one model, with the x\-axis showing model size in billions of parameters on a log scale and the y\-axis showing average SPSS across datasets\. The plot suggests that prompt sensitivity does not decrease monotonically with model size\. Although the 70B model has the lowest average SPSS, models in the 7B–14B range vary substantially, indicating that model scale alone does not explain sycophancy prompt sensitivity\.
## Appendix BAdditional Prompt\-wise Results

We provide full prompt\-wise profiles in Appendix[J](https://arxiv.org/html/2608.23837#A10)\. These profiles complement the main heatmap in Figure[3](https://arxiv.org/html/2608.23837#S6.F3)by showing the full trend view across all prompt variants for each dataset and model\.

## Appendix CSmall Paraphrase Robustness Check

To test whether the main directional effects depend on a single lexical realization of each prompt cue, we conduct a small paraphrase robustness check for the three most theoretically salient prompt types: validation\-seeking, emotional pressure, and anti\-sycophancy\. We run this check on a subset of SS and AITA\-YTA items using the same response parsing and sycophancy labels as in the main experiments\. For each cue type, we compare the original prompt wording with two paraphrases, and compute directional effects relative to the neutral prompt P0\.

Table 2:Paraphrased social\-cue wordings used in the small paraphrase robustness check\. Each paraphrase is inserted into the same dataset\-specific prompt template as the corresponding original cue\.The purpose of this check is not to exhaustively cover all possible wordings, but to verify whether the qualitative direction of the main effects is stable under nearby lexical realizations\. Table[3](https://arxiv.org/html/2608.23837#A3.T3)compares the original directional effect with the mean effect across the two paraphrases for each dataset and cue type\.

Table 3:Small paraphrase robustness check\. Directional effects are computed relative to the neutral prompt P0\. Sign consistency indicates whether the paraphrase\-mean effect has the same direction as the original prompt cue\.
## Appendix DBootstrap Confidence Intervals

We compute bootstrap confidence intervals using item\-level non\-parametric bootstrap resampling with 1,000 bootstrap samples\. For each bootstrap sample, items are resampled with replacement while preserving all prompt variants for each sampled item\.

Table 4:Representative bootstrap confidence intervals for BaseSyc and SPSS\. Intervals are computed by item\-level bootstrap resampling with 1,000 samples\.
## Appendix EOEQ Judge Validation Details

We provide additional details for the manual validation of the GPT\-4\-based judge used to label OEQ responses\.

#### Annotation guideline\.

Annotators label a response asSycophanticif it uncritically endorses, validates, or morally affirms the user’s framing, especially when it accepts the user’s perspective without meaningful qualification or independent assessment\. Annotators label a response asNon\-Sycophanticif it provides a balanced, independent, corrective, or qualified assessment, even if it expresses empathy toward the user\. Empathy alone is not treated as sycophancy unless it also validates the user’s framing or conclusion\. Annotators useUnclearwhen they cannot confidently determine whether the response is sycophantic\.

#### Sampling details\.

The validation set consists of 500 OEQ model responses sampled from the full set of generated OEQ outputs\. The sampling unit is a single model response to one OEQ item under one prompt variant\. The sample is stratified across models, prompt variants, and GPT\-4 judge\-predicted labels where possible\. The validation set is balanced between judge\-predictedSycophanticandNon\-Sycophanticlabels where possible\.

#### Agreement metrics\.

After manual annotation, we compare GPT\-4 judge labels against manual labels after excludingUnclearcases\. The GPT\-4\-based judge achieves 86\.4% agreement with manual annotations and Cohen’sκ=0\.73\\kappa=0\.73, indicating substantial agreement\. TheUnclearrate is 7\.2%\.

#### GPT\-4 judge prompt\.

For OEQ, we use the following GPT\-4\-based judging prompt to assign binary sycophancy labels to open\-ended model responses\.

> You are evaluating whether an assistant response is sycophantic\. A response isSycophanticif it uncritically agrees with, validates, endorses, or morally affirms the user’s framing, especially when it accepts the user’s perspective without meaningful qualification or independent assessment\. A response isNon\-Sycophanticif it provides a balanced, independent, corrective, or qualified assessment, even if it expresses empathy toward the user\. Empathy alone should not be treated as sycophancy unless the response also validates the user’s framing or conclusion\. If the response is ambiguous or cannot be confidently classified, outputUnclear\. Given the user situation and the assistant response, output exactly one label:Sycophantic,Non\-Sycophantic, orUnclear\. User situation: \{prompt\} Assistant response: \{response\}

We mapSycophanticto 1 andNon\-Sycophanticto 0\.Unclearcases are manually inspected and excluded only when no reliable sycophancy label can be assigned\.

#### Common disagreement patterns\.

We qualitatively inspect judge–human disagreement cases\. The most common disagreements occur when a response expresses empathy or acknowledges the user’s feelings without clearly endorsing the user’s framing, or when a response combines partial validation with later qualification\. These cases illustrate the difficulty of separating appropriate emotional support from sycophantic affirmation in open\-ended advice responses\.

## Appendix FDataset and Prompt Examples

This section corresponds to Table 5\.

Table 5:Representative dataset examples and prompt variants\. For SS, sycophancy is operationalized as anAgreeresponse to an assumption\-laden statement\. For OEQ, open\-ended responses are labeled by a GPT\-4\-based judge\. For AITA\-YTA, sycophancy is operationalized as anNTAverdict because the examples are drawn from a subset whose original community judgment isYTA\.
## Appendix GPrompt\-Induced Label Change Examples

This section corresponds to Table 6\.

Table 6:Representative prompt\-induced sycophancy label changes\. These examples illustrate cases where changing the user’s framing cues changes the model’s final stance or verdict for the same underlying item\. We focus on AITA\-YTA and SS because their output labels are directly observable\. For OEQ, responses are open\-ended and evaluated by a GPT\-4\-based judge; in the current results, OEQ is analyzed through aggregate sycophancy rates and directional prompt effects rather than discrete flip examples\.
## Appendix HPrompt Templates

This section corresponds to Tables 7–10\. We use eight controlled prompt variants for each dataset\. Table[7](https://arxiv.org/html/2608.23837#A8.T7)summarizes the framing cue used for each prompt variant, and Tables[8](https://arxiv.org/html/2608.23837#A8.T8)–[10](https://arxiv.org/html/2608.23837#A8.T10)provide the full dataset\-specific templates including answer\-format instructions\.

Table 7:Controlled prompt variants used across datasets\. For SS, each variant modifies an assumption\-laden statement before asking forAgreeorDisagree\. For OEQ, each variant modifies the user\-side social cue before asking for an open\-ended advice response, which is later labeled by a GPT\-4\-based judge\. For AITA\-YTA, each variant modifies the user\-side social cue before asking for aYTAorNTAjudgment\. For P0, we show the main task instruction; for P1–P7 we show only the framing prefix for P1–P6 or appended line for P7 for compactness\.Table 8:Full prompt templates for the SS dataset\. The placeholder \{sentence\} denotes the assumption\-laden subjective statement\.Table 9:Full prompt templates for the OEQ dataset\. The placeholder \{prompt\} denotes the open\-ended advice query\.Table 10:Full prompt templates for the AITA\-YTA dataset\. The placeholder \{prompt\} denotes the AITA post\.
## Appendix IDirectional Prompt Effects

This section corresponds to Figure[6](https://arxiv.org/html/2608.23837#A9.F6)\.

![Refer to caption](https://arxiv.org/html/2608.23837v1/figure3_prompt_profile_main.png)Figure 6:Prompt effects relative to the neutral prompt P0\. Each panel reports the change in mean sycophancy proxy score relative to P0\. Positive values indicate increased sycophancy, while negative values indicate reduced sycophancy\.
## Appendix JFull Prompt\-wise Profiles

This section corresponds to Figure[7](https://arxiv.org/html/2608.23837#A10.F7)\.

![Refer to caption](https://arxiv.org/html/2608.23837v1/figure4_prompt_profile_appendix.png)Figure 7:Full prompt\-wise sycophancy profiles across all prompt variants\. Each panel reports the absolute dataset\-specific sycophancy proxy score under each prompt variant\. For SS, the proxy is theAgreerate; for OEQ, it is the GPT\-4\-judge\-labeled sycophancy rate; and for AITA\-YTA, it is theNTArate\. These profiles complement the main heatmap in Figure[3](https://arxiv.org/html/2608.23837#S6.F3)by providing the full prompt\-wise trend view for each dataset and model\.