Paper page - Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

Source: https://huggingface.co/papers/2605.17360

Abstract

A benchmark and evaluation framework for real-time duplex interaction in multimodal large language models, assessing continuous response generation and proactive event detection in streaming scenarios.

Real-time duplex interactionis essential for multimodal AI systems operating in real-world scenarios, where models must continuously process streaming inputs and respond at appropriate moments. However, most existingmultimodal large language models(MLLMs) are evaluated in offline settings, where the entire video input is processed before any response is generated. While recent work has started to explore real-time duplex MLLMs, there is still no comprehensivebenchmarkorautomatic evaluationmethod for this setting. To address this gap, we propose Omni-DuplexEval, abenchmarkfor systematically evaluatingreal-time duplex interaction. Thebenchmarkconsists of two complementary scenarios: (1) Real-Time Description, which evaluates the ability to generate continuous, time-aligned responses that track evolving multimodal inputs, and (2) Proactive Reminder, which evaluates the ability to identify salient events and respond at appropriate moments. Omni-DuplexEval contains 660 videos with fine-grained, human-annotated labels and precise temporal metadata, spanning 9 tasks grounded in real-world scenarios, where all questions are formulated as open-ended queries. We further introduce anautomatic evaluationframework based onLLM-as-a-Judge, which enables systematic assessment by jointly evaluatingresponse-content alignmentandresponse timingthrough timestamp-aware andsequential reasoning, achieving strong alignment with human judgments. Experiments on state-of-the-art duplex MLLMs reveal substantial limitations. The best-performing model achieves only 39.6% overall, while scoring only 20.0% on Proactive Reminder. Our analysis identifies two key challenges: models struggle to balance timely responses with coherent, holistic content generation, and they often fail to determine both when to respond and what to produce. We hope our work facilitates further progress in MLLMs.

View arXiv pageView PDFGitHub2Add to collection

Get this paper in your agent:

hf papers read 2605\.17360

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.17360 in a model README.md to link it from this page.

Datasets citing this paper1

#### Hothan/Omni-DuplexEval Viewer• Updated16 minutes ago • 660 • 206

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.17360 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.