Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?
Summary
The paper introduces SpatialUncertain, a benchmark to evaluate whether vision-language models recognize when they cannot answer spatial questions due to occlusion or perspective ambiguity, revealing overconfidence and poor abstention behavior.
View Cached Full Text
Cached at: 06/01/26, 07:18 AM
Paper page - Seeing Isn’t Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?
Source: https://huggingface.co/papers/2605.30557
Abstract
Vision-language models exhibit overconfidence in spatial reasoning tasks and struggle to identify when additional observations are needed to resolve uncertainty.
Spatial reasoningis a fundamental capability forvision-language models(VLMs) deployed in real-world environments. However, visual observations are inherently limited representations of a 3D world:occlusioncan render objects invisible, and perspective can make geometric properties misleading. Despite this, existingspatial reasoningbenchmarks typically assume that observations are sufficient and reliable, focusing on whether models produce correct answers rather than whether they recognize when a question cannot be answered and what additional observations would be needed. In this work, we challenge this assumption by constructing a controlled evaluation framework, SpatialUncertain, and introducing two types of observation challenges: (1)occlusion, which hides target information, and (2)perspective ambiguity, which produces misleading visual cues. For each configuration, we design spatial questions that are answerable under clean observations but requireabstentionunder the introduced challenges. We further evaluate whether models can identify whichadditional viewpointswould resolveperspective ambiguity. Our results across a diverse set of frontier open- and closed-source VLMs reveal two consistent failure modes. First, models are prone to overconfident answering, attempting to solvespatial reasoningtasks even whenvisual evidenceis incomplete or misleading, with average accuracy around 30\% underocclusionand below 10\% underperspective ambiguity. Second, even when additional views are available, some models perform near random chance in identifying which would provide reliable evidence. Together, our findings call for moving beyond answer correctness toward evaluating whether models know when to abstain and how to seek reliable evidence.
View arXiv pageView PDFProject pageGitHub3Add to collection
Get this paper in your agent:
hf papers read 2605\.30557
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.30557 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.30557 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.30557 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
This paper introduces Act2Answer, a protocol to evaluate knowledge retention in Vision-Language-Action (VLA) models by requiring agents to answer questions through physical actions. It finds that VLAs retain basic knowledge but show gaps on richer semantic categories, and that VQA co-training helps.
Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
This paper investigates a bias in vision-language models where they overestimate shared understanding in dialogue, confusing perceptual access with communicative grounding. The findings have implications for dialogue systems and VLM evaluation.
SPACENUM: Revisiting Spatial Numerical Understanding in VLMs
This paper presents SpaceNum, a unified framework to evaluate how vision-language models (VLMs) understand numerical values in spatial contexts, finding that current models largely fail to ground numbers spatially and often perform close to random guessing.
Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
This paper investigates whether vision-language models can distinguish potential from established common ground in asymmetric dialogue. Experiments on MapTask data show that providing task-relevant map content (visual or textual) biases models toward over-predicting alignment, as they rely on static referential cues rather than tracking grounding through dialogue history.
When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understanding
This paper studies the ability of multimodal large language models (MLLMs) to detect when the correct answer is absent in video understanding tasks, finding that models systematically fail by selecting plausible distractors instead of recognizing no valid option exists. The failure worsens in temporal reasoning and dense frame sampling, and chain-of-thought prompting only partially mitigates the issue.