Video-Oasis: Rethinking Evaluation of Video Understanding

Hugging Face Daily Papers Papers
video-understanding benchmark diagnostic-suite video-llm computer-vision evaluation

Summary

Video-Oasis reveals that 55% of existing video benchmarks can be solved without visual input, exposing significant capability gaps in current video understanding models. State-of-the-art models perform only marginally above random guessing on the remaining video-native challenges.

The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.
Original Article
View Cached Full Text

Cached at: 07/10/26, 06:15 AM

Paper page - Video-Oasis: Rethinking Evaluation of Video Understanding

Source: https://huggingface.co/papers/2603.29616

Abstract

Video-Oasis diagnostics reveal that half of existing video benchmarks can be solved without visual input, exposing significant capability gaps in current video understanding models.

The inherent complexity ofvideo understandingmakes it difficult to determine whetherVideo-LLMbenchmark performance stems fromvisual perception,linguistic reasoning, orknowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluatingvideo understandingremain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluatingvideo understanding. In this work, we introduce Video-Oasis, a sustainablediagnostic suitefor systematically auditing existingvideo understandingbenchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remainingvideo-native challengesexpose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate whichalgorithmic design choicescontribute to robustvideo understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating futureVideo-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.

View arXiv pageView PDFProject pageGitHub14Add to collection

Get this paper in your agent:

hf papers read 2603\.29616

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2603.29616 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2603.29616 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2603.29616 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding

Hugging Face Daily Papers

VideoKR introduces a large-scale video reasoning dataset and benchmark designed to enhance knowledge-intensive video understanding through expert-domain content and human-in-the-loop example generation. The dataset contains 315K video reasoning examples over 145K expert-domain videos.

OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs

Papers with Code Trending

OVO-S-Bench introduces a comprehensive human-annotated benchmark of 1,680 questions across 348 videos to evaluate streaming spatial intelligence in multimodal LLMs, revealing that even the best model (Gemini-3.1-Pro) trails human experts by 27 points. The benchmark exposes key limitations including allocentric mapping as a major bottleneck and chain-of-thought reasoning amplifying spatial errors.