@PinzhiHuang: State tracking is a core pillar of video understanding: it requires identifying entities and events, and mapping how th…
Summary
Introduces VSTAT, a new benchmark to measure how well multimodal LLMs track states in videos, revealing that frontier models struggle with tasks humans find easy.
View Cached Full Text
Cached at: 06/03/26, 03:42 AM
State tracking is a core pillar of video understanding: it requires identifying entities and events, and mapping how their states evolve over time.
Frontier multimodal models are surprisingly bad at it, so we built a benchmark to measure it.
Meet VSTAT!
Paper: https://arxiv.org/abs/2606.03920 Website: https://vision-x-nyu.github.io/vstat-site/ Github: https://github.com/vision-x-nyu/vstat… HF: https://huggingface.co/datasets/nyu-visionx/vstat…
I’m grateful to work with such amazing collaborators @sihyun_yu and @ma_nanye, as well as @hyunseok_i @shushengyang @ellisbrown @ojmichel4 @boyangzheng @JinwooShin0417!!!
Thank you to @sainingxie for advising this project AND also cooking up a fun VSTAT task!
Similar Articles
Benchmarking Visual State Tracking in Multimodal Video Understanding
Introduces VSTAT, a benchmark for evaluating visual state tracking in multimodal large language models (MLLMs) using 834 clips and 1,500 questions. Current MLLMs perform poorly compared to humans, failing at visual perception rather than reasoning.
@ma_nanye: VSTAT highlights the substantial perceptual gap between humans and MLLMs, but it goes far beyond that. Its diverse task…
VSTAT is a new benchmark for visual state tracking in videos that reveals perceptual gaps between humans and multimodal LLMs.
StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models
StateSight is a new benchmark for evaluating spatial-state reconstruction in vision-language models, showing that models like GPT-5.5 and Claude Sonnet 5 struggle with spatial reasoning tasks compared to human performance.
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
A survey presenting a human-view perspective on video understanding with multimodal large language models, organized around watching, remembering, and reasoning abilities, covering challenges, methods, and applications.
LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
LLaVA-OneVision-2 introduces codec-stream tokenization and windowed attention for efficient video understanding, achieving state-of-the-art performance across multiple multimodal benchmarks including video, spatial, and tracking tasks.