Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue

Hugging Face Daily Papers Papers
vision-language-models dialogue-systems common-ground evaluation asymmetric-dialogue bias grounded-language-understanding

Summary

This paper investigates a bias in vision-language models where they overestimate shared understanding in dialogue, confusing perceptual access with communicative grounding. The findings have implications for dialogue systems and VLM evaluation.

In collaborative dialogue, shared perception does not guarantee shared interpretation. Mutual understanding must be established through interaction. We investigate whether vision-language models (VLMs) can distinguish what could be shared from what has been shared between dialogue participants through grounding. We formulate this as an interpretation-matching task on 13,077 annotated reference expressions from HCRC MapTask dialogues, and evaluate VLMs under systematically controlled manipulations of dialogue context and map-information access. Our results show that providing authentic map images improves overall performance but shifts models toward over-predicting alignment. Textual descriptions of the same map content reproduce this bias, while non-informative images suppress alignment predictions entirely, indicating that the bias is driven by task-relevant map content, not the visual channel. This improvement comes at the cost of degraded accuracy on non-aligned cases. Calibration analysis and reference-chain tracking further suggest that models rely on static referential cues on the maps rather than tracking how grounding unfolds through dialogue history. We observe these patterns most clearly in Qwen3-VL-8B-Instruct and, to varying degrees, in four additional models from two architecture families. In models that exhibit the bias, map content, whether presented visually or textually, is treated as evidence of mutual understanding, conflating potential with established common ground.
Original Article
View Cached Full Text

Cached at: 07/03/26, 03:52 AM

Paper page - Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue

Source: https://huggingface.co/papers/2606.31719 This paper investigates a subtle but important distinction in collaborative dialogue: whether vision-language models can tell apart what could be shared (from shared perception) versus what has been shared (through grounding in interaction). Using 13,077 annotated reference expressions from HCRC MapTask dialogues, we evaluate VLMs under controlled manipulations of dialogue context and map-information access.

A key finding is that providing authentic map images improves overall VLM performance but introduces a systematic bias toward over-predicting alignment between participants — models tend to assume interlocutors share the same interpretation simply because they share the same visual input. Interestingly, textual descriptions of the same map content reproduce this bias, while non-informative images suppress alignment predictions entirely, suggesting the bias is driven by task-relevant content rather than the visual modality itself.

This has implications for anyone working on dialogue systems, grounded language understanding, or VLM evaluation: current models conflate perceptual access with communicative grounding, which is precisely the kind of error that matters in real collaborative settings. We’d be curious to hear thoughts on how this bias might be mitigated — whether through training objectives that explicitly model asymmetric information states, or through architectural changes that separate perceptual and discourse-level representations.

Similar Articles

Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models

arXiv cs.CL

This paper investigates how vision-language models resolve conflicts between visual evidence and world knowledge, revealing that visual grounding is the default while prior knowledge depends on a small set of late-layer attention heads. The authors perform causal analysis across three VLM families, demonstrating an asymmetric structure where ablating these heads shifts predictions from knowledge-grounded to visually grounded answers.