OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains

Hugging Face Daily Papers Papers
audio-visual question-answering dataset multimodal reasoning automated-pipeline

Summary

OmniVideo-100K introduces an automated data engine with entity-anchored scripting and clue-guided QA generation to improve audio-visual reasoning and temporal consistency, achieving significant performance gains across multiple benchmarks.

Current automated pipelines for audio-visual Question Answering (QA) generally adopt a ``video-caption-QA'' paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning. To address these issues, we propose an automated data engine featuring two mechanisms: (1) Entity-Anchored Video Scripting transforms videos into structured scripts, comprising summaries, main entity lists, and segment-wise audio-visual descriptions. The entity list serves as a global prior to ensure cross-segment referential consistency and reconstruct audio-visual associations. (2) Clue-Guided QA Generation prompts models to first mine cross-segment, multimodal clues from the script, and subsequently generate QA pairs based on these high-value clues. Leveraging this pipeline, we construct the instruction-tuning dataset OmniVideo-100K and a human-verified test set, OmniVideo-Test. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B and Qwen3-Omni-30B on OmniVideo-100K yields performance gains of up to 20.59% on OmniVideo-Test, demonstrating strong generalization (up to 12.64% improvements) across established benchmarks like Daily-Omni and JointAVBench.
Original Article
View Cached Full Text

Cached at: 06/15/26, 09:05 AM

Paper page - OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains

Source: https://huggingface.co/papers/2606.14702

Abstract

An automated audio-visual question answering system uses entity-anchored video scripting and clue-guided QA generation to improve cross-modal reasoning and temporal consistency in video analysis.

Current automated pipelines foraudio-visual Question Answering(QA) generally adopt a ``video-caption-QA’’ paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning. To address these issues, we propose an automated data engine featuring two mechanisms: (1) Entity-Anchored Video Scripting transforms videos into structured scripts, comprising summaries, main entity lists, and segment-wise audio-visual descriptions. The entity list serves as a global prior to ensure cross-segment referential consistency and reconstruct audio-visual associations. (2) Clue-Guided QA Generation prompts models to first mine cross-segment,multimodal cluesfrom the script, and subsequently generate QA pairs based on these high-value clues. Leveraging this pipeline, we construct theinstruction-tuning datasetOmniVideo-100Kand a human-verified test set,OmniVideo-Test. Fine-tuningVITA-1.5,Qwen2.5-Omni-7BandQwen3-Omni-30BonOmniVideo-100Kyields performance gains of up to 20.59% onOmniVideo-Test, demonstrating strong generalization (up to 12.64% improvements) across established benchmarks likeDaily-OmniandJointAVBench.

View arXiv pageView PDFProject pageGitHub10Add to collection

Get this paper in your agent:

hf papers read 2606\.14702

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.14702 in a model README.md to link it from this page.

Datasets citing this paper2

#### MiG-NJU/OmniVideo-100K Preview• Updatedabout 7 hours ago #### MiG-NJU/OmniVideo-Test Viewer• Updatedabout 7 hours ago • 505 • 1

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.14702 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

Hugging Face Daily Papers

OmniReasoning introduces OmniReasoningBench, a data engine (OmniQA) producing SFT and RL training sets, and a Modality-Factored Self-Distillation method, yielding a 30B-A3B model that outperforms Qwen3-Omni-30B-A3B-Thinking by double-digit margins on audio-visual joint reasoning benchmarks.

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

Hugging Face Daily Papers

OmniSeek is an agentic framework that turns an Omni-LLM into an active multi-turn reasoning agent with native tool use, dynamically retrieving audio/visual evidence across long contexts. It uses a synthetic OmniTraj-170K trajectory corpus, supervised cold-starting, and two-stage verifiable-reward RL with an Audio-Visual Necessity objective to boost cross-modal reasoning.

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

Hugging Face Daily Papers

This paper introduces OmniScript, an 8B-parameter omni-modal (audio-visual) language model for a novel video-to-script (V2S) task that generates hierarchical, scene-by-scene scripts from long-form cinematic videos. Trained via progressive pipeline techniques including chain-of-thought SFT and reinforcement learning with temporally segmented rewards, OmniScript outperforms larger open-source models and rivals proprietary models like Gemini 3-Pro.

VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding

Hugging Face Daily Papers

VideoKR introduces a large-scale video reasoning dataset and benchmark designed to enhance knowledge-intensive video understanding through expert-domain content and human-in-the-loop example generation. The dataset contains 315K video reasoning examples over 145K expert-domain videos.

A Very Big Video Reasoning Suite

Papers with Code Trending

This paper introduces the Very Big Video Reasoning (VBVR) dataset and benchmark, a large-scale resource with over one million video clips across 200 reasoning tasks, enabling systematic study of spatiotemporal reasoning and showing early signs of emergent generalization.