Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

Hugging Face Daily Papers Papers
video-understanding multimodal-agent long-term-memory reflex-reasoning efficient-video-processing hierarchical-merging parametric-latent-state

Summary

Light-Omni is a multimodal agent framework for efficient video understanding that uses dual contextual states (global state and parametric latent state) to avoid iterative reasoning, achieving faster and more accurate processing with significant speedup and memory savings.

Agentic video understanding equips models with long-term memory to autonomously process and respond to continuous, long-horizon multimodal streams. However, advanced video agents often rely on ``detective-style'' iterative reasoning for action control (e.g., search) and evidence aggregation, incurring prohibitive costs and latency. We argue that such heavy reasoning primarily compensates for the lack of global context and semantic misalignment in retrieval. This paper introduces Light-Omni, a multimodal agent framework for reflexive and lightweight video understanding. It achieves this through dual contextual states that instantly build the required context in a single forward pass. First, we maintain a global state, a finite-sized multimodal script continuously consolidated from episodic memory, serving as the global context for Light-Omni. Through hierarchical merging, it preserves recent details while summarizing past events. Second, conditioned on this global context, we generate a parametric latent state that directly drives autonomous actions and produces retrieval embeddings, with minimal latency. Benefiting from this coupled design, Light-Omni achieves semantically aligned retrieval and reflexive responses while avoiding iterative reasoning. Extensive experiments validate the effectiveness of Light-Omni across multiple video benchmarks. Notably, it outperforms M3-Agent with an average 2.4% accuracy gain, a 12.1times speedup, and a 2.6times improvement in GPU memory efficiency. Furthermore, it serves as a memory system to enhance both the performance and efficiency of existing MLLMs. Project page: https://clare-nie.github.io/Light-Omni.
Original Article
View Cached Full Text

Cached at: 07/08/26, 02:47 AM

Paper page - Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

Source: https://huggingface.co/papers/2607.05511

Abstract

Light-Omni is a multimodal agent framework that enables efficient video understanding through dual contextual states, achieving faster and more accurate video processing by eliminating iterative reasoning while maintaining semantic alignment.

Agentic video understanding equips models with long-term memory to autonomously process and respond to continuous, long-horizon multimodal streams. However, advanced video agents often rely on ``detective-style’’iterative reasoningfor action control (e.g., search) and evidence aggregation, incurring prohibitive costs and latency. We argue that such heavy reasoning primarily compensates for the lack of global context and semantic misalignment in retrieval. This paper introduces Light-Omni, amultimodal agent frameworkfor reflexive and lightweight video understanding. It achieves this through dual contextual states that instantly build the required context in a single forward pass. First, we maintain aglobal state, a finite-sized multimodal script continuously consolidated fromepisodic memory, serving as the global context for Light-Omni. Throughhierarchical merging, it preserves recent details while summarizing past events. Second, conditioned on this global context, we generate aparametric latent statethat directly drives autonomous actions and producesretrieval embeddings, with minimal latency. Benefiting from this coupled design, Light-Omni achieves semantically aligned retrieval and reflexive responses while avoidingiterative reasoning. Extensive experiments validate the effectiveness of Light-Omni across multiplevideo benchmarks. Notably, it outperforms M3-Agent with an average 2.4% accuracy gain, a 12.1times speedup, and a 2.6times improvement in GPU memory efficiency. Furthermore, it serves as a memory system to enhance both the performance and efficiency of existingMLLMs. Project page: https://clare-nie.github.io/Light-Omni.

View arXiv pageView PDFProject pageGitHub17Add to collection

Get this paper in your agent:

hf papers read 2607\.05511

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### ClareNie/Light-Omni Image-Text-to-Text• 12B• Updatedabout 2 hours ago • 4 • 5

Datasets citing this paper1

#### ClareNie/Light-Omni-Training Viewer• Updatedabout 1 hour ago • 511k • 71 • 2

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles