Benchmarking Visual State Tracking in Multimodal Video Understanding
Summary
Introduces VSTAT, a benchmark for evaluating visual state tracking in multimodal large language models (MLLMs) using 834 clips and 1,500 questions. Current MLLMs perform poorly compared to humans, failing at visual perception rather than reasoning.
View Cached Full Text
Cached at: 06/03/26, 03:35 AM
Paper page - Benchmarking Visual State Tracking in Multimodal Video Understanding
Source: https://huggingface.co/papers/2606.03920 Authors:
,
,
,
,
,
,
,
,
,
Abstract
Current multimodal large language models struggle with visual state tracking in videos, performing poorly even when human-level capabilities are required, and existing agentic approaches do not effectively address these limitations.
Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity forvisual state trackingis fundamental tovideo understanding, yet remains underexplored in current evaluations ofMultimodal Large Language Models(MLLMs). We introduceVisual STAte Trackingbenchmark (VSTAT), a video-based benchmark designed to diagnosevisual state trackingin MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiringcontinuous perceptionand integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs’ thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-basedvideo agentsandcoding agents, do not readily resolve these failures, still falling short on VSTAT.
View arXiv pageView PDFProject pageGitHub3Add to collection
Get this paper in your agent:
hf papers read 2606\.03920
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.03920 in a model README.md to link it from this page.
Datasets citing this paper1
#### nyu-visionx/vstat Viewer• Updatedabout 1 hour ago • 530 • 495 • 1
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.03920 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
@PinzhiHuang: State tracking is a core pillar of video understanding: it requires identifying entities and events, and mapping how th…
Introduces VSTAT, a new benchmark to measure how well multimodal LLMs track states in videos, revealing that frontier models struggle with tasks humans find easy.
@ma_nanye: VSTAT highlights the substantial perceptual gap between humans and MLLMs, but it goes far beyond that. Its diverse task…
VSTAT is a new benchmark for visual state tracking in videos that reveals perceptual gaps between humans and multimodal LLMs.
OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs
OVO-S-Bench introduces a comprehensive human-annotated benchmark of 1,680 questions across 348 videos to evaluate streaming spatial intelligence in multimodal LLMs, revealing that even the best model (Gemini-3.1-Pro) trails human experts by 27 points. The benchmark exposes key limitations including allocentric mapping as a major bottleneck and chain-of-thought reasoning amplifying spatial errors.
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
A survey presenting a human-view perspective on video understanding with multimodal large language models, organized around watching, remembering, and reasoning abilities, covering challenges, methods, and applications.
ViMU: Benchmarking Video Metaphorical Understanding
ViMU is the first benchmark designed to evaluate video understanding models' ability to interpret metaphorical, ironic, and social meanings beyond literal visual comprehension, using hint-free open-ended and multiple-choice questions.