OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Hugging Face Daily Papers Papers
audio-visual-dialogue multimodal-learning reinforcement-learning benchmark data-synthesis omni-model

Summary

This paper introduces OmniVChat, a task for native audio-visual dialogue, and presents a data engine, benchmark, and reinforcement learning method to train and evaluate omni models, demonstrating improvements on synthesized and human-recorded data.

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Original Article
View Cached Full Text

Cached at: 09/21/26, 11:20 AM

Paper page - OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Source: https://huggingface.co/papers/2609.21465 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

WedefineOmniVChat(OmniVideoChat)asthetaskofnativeaudio-visualdialoguebetweenauserandanomnimodel.InOmniVChat,omnimodelsdirectlyandsimultaneouslyreceiveaudioandvideofromauserandreturntext.Theuser’squeryisembeddedintheaudioandvideo,withoutaseparatetextquestion,externalcaptioning,orspeechrecognition.Directaudio-visualinputreducesexternallatencyandcomputationwhilepreservingperceptualcues.However,researchonOmniVChatfacestwoconstraints:dataavailabilityandevaluation.Recordingsofpeopleusingtheirowndevicesarescarce.Furthermore,agoodreplyoftenneedstoaccountfortheuser’ssurroundings,facialexpressions,andnearbyobjects,andsuchresponsescanbeexpressedinmanydifferentways,makingkeywordmatchingunreliableforevaluatingreplyquality.Recentprogressinagentsystemsandvideogenerationmakesgenerationforcomprehensionviable,whichmeansusingsynthesizeddialoguesfortrainingandevaluation.Therefore,wepresentOmniVChat-Studio,amulti-agentdataengineforsynthesizingsingle-andmulti-turnaudio-visualdialogues.WeusesynthesizeddialoguestobuildOmniVChat-Bench,anevaluationbenchmarkthatevaluatesomnimodels’basicdialogueabilitiesacrossfiveabilitycategories.WealsopresentOmniVChat-RL,areinforcementlearningrewarddesignthatjointlytargetsreplycorrectness,efficiency,andstyleinOmniVChat.TrainingQwen3-Omni-InstructwithOmniVChat-RLonsynthesizeddialoguesimprovesitsperformanceonbothOmniVChat-Benchandthehuman-recordedOmniVChat-Bench-Human.Thesegainsvalidatetherewarddesignandshowtransfertoreal-worlddialoguesintrainingandevaluation.

View arXiv pageView PDFGitHub10Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.21465 in a model README.md to link it from this page.

Datasets citing this paper1

#### Harland/OmniVChat Viewer• Updatedabout 6 hours ago • 2.8k • 68 • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.21465 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

Hugging Face Daily Papers

This paper introduces Omni-DuplexEval, a benchmark and automatic evaluation framework for real-time duplex interaction in multimodal large language models, assessing continuous response generation and proactive event detection in streaming scenarios.