Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks
Summary
This paper introduces an information-asymmetric spot-the-difference task to measure epistemic vigilance in vision-language models, finding that models often overlook private evidence to agree with partners. Model steering to reduce sycophancy improves reliability in cooperative tasks.
View Cached Full Text
Cached at: 08/03/26, 07:37 AM
# Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks Source: [https://arxiv.org/abs/2607.29585](https://arxiv.org/abs/2607.29585) [View PDF](https://arxiv.org/pdf/2607.29585) > Abstract:To maintain common ground in cooperative conversation, humans iteratively update their beliefs as conversation participants share new information; participants who are epistemically vigilant detect when new information conflicts with prior beliefs and take steps to repair these conflicts\. In order for AI systems to serve as reliable partners in complex cooperative tasks, they must similarly weigh incoming information against their own private evidence and shared context and appropriately surface inconsistencies when they arise\. To measure the epistemic vigilance of vision\-language models in cooperative settings, we present an information\-asymmetric, dialog\-based "spot\-the\-difference" task\. Two models are privately shown one image each, and must determine through conversation whether the images are identical or, if not, identify the difference\. Models routinely fail at this: they frequently overlook key evidence in their private image in favor of agreeing with their conversational partner, even when their agreement is unwarranted\. We relate these violations of epistemic vigilance to the broader behavior of sycophancy, which manifests itself in cooperative goal\-oriented dialog as over\-accommodation and weak evidential grounding\. Our results show that model steering to reduce sycophancy with a vector learned from task\-agnostic sycophancy examples can reduce epistemic vigilance\-related errors, making models more faithful reporters of their evidence, and in turn, more reliable partners in information\-asymmetric cooperative tasks\. ## Submission history From: Rupak Sarkar \[[view email](https://arxiv.org/show-email/ee3c05b4/2607.29585)\] **\[v1\]**Fri, 31 Jul 2026 16:09:23 UTC \(2,622 KB\)
Similar Articles
Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
This paper investigates whether vision-language models can distinguish potential from established common ground in asymmetric dialogue. Experiments on MapTask data show that providing task-relevant map content (visual or textual) biases models toward over-predicting alignment, as they rely on static referential cues rather than tracking grounding through dialogue history.
Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
This paper investigates a bias in vision-language models where they overestimate shared understanding in dialogue, confusing perceptual access with communicative grounding. The findings have implications for dialogue systems and VLM evaluation.
Dual-Stance Evaluation of Sycophancy: The Structure of Agreement and the Limits of Intervention
This paper introduces dual-stance evaluation to test whether activation steering for reducing sycophancy also suppresses agreement with factually correct statements, finding that the steering direction cannot differentially target sycophantic vs factual agreement.
When Helpfulness Becomes Sycophancy: Sycophancy is a Boundary Failure Between Social Alignment and Epistemic Integrity in Large Language Models
This position paper analyzes sycophancy in LLMs as a boundary failure between social alignment and epistemic integrity, proposing a new framework and taxonomy to classify and mitigate these behaviors.
Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
This paper introduces CoT-Mediate, a behavioral framework to test whether chain-of-thought reasoning in medical vision-language models actually drives predictions or merely decorates them. Auditing LLaVA-Med and MedGemma on VQA-RAD, it finds that how reasoning is injected (prefix-forcing vs re-prompting) and the attributed source (self vs expert) significantly affect model faithfulness and sycophancy.