LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
Summary
LatentOmni proposes a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states, outperforming explicit text-based chain-of-thought methods in audio-visual reasoning tasks.
View Cached Full Text
Cached at: 05/22/26, 06:27 AM
Paper page - LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
Source: https://huggingface.co/papers/2605.22012 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
LatentOmni is a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states using feature-level supervision and temporal consistency embedding, outperforming explicit text-based chain-of-thought approaches in audio-visual reasoning tasks.
Jointaudio-visual reasoningis essential for omnimodal understanding, yet currentmultimodal large language models(MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-basedchain-of-thought(CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unifiedlatent spaceis a better medium for such reasoning because it preserves densesensory informationwhile remaining compatible withautoregressive generation. Based on this insight, we propose LatentOmni, across-modal reasoningframework that interleaves textual reasoning with audio-visual latent states. LatentOmni introducesfeature-level supervisionto align latent reasoning states with task-relevant sensory features and usesOmni-Sync Position Embedding(OSPE) to maintaintemporal consistencybetween latent audio and visual states. We further construct LatentOmni-Instruct-35K, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multipleaudio-visual reasoningbenchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.22012
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.22012 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.22012 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.22012 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
FaithformBench: Benchmarking Faithfulness of Mathematical Chain-of-Thought Autoformalisation
Introduces FaithformBench, a benchmark for assessing the faithfulness of mathematical chain-of-thought autoformalisation systems by measuring validity and invalidity preservation on perturbed steps. Applied to eight AF systems, it reveals widespread sycophancy where invalid inputs are silently corrected.
When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning
This empirical study tests when chain-of-thought prompting helps or hurts LLM reasoning, finding that CoT provides large gains on deep serial tasks like GSM8K and MATH but is redundant on shallow tasks like MMLU and ARC-Challenge, consistent with a serial-depth bottleneck framework.
Actionable Hallucination Detection: Translating Latent Uncertainty into Agentic Critique
This paper introduces the Latent Critic, a lightweight LoRA adapter that detects hallucinated agent actions in real time by restructuring the transformer's residual stream into localized natural-language feedback, achieving 0.966 AUROC and enabling self-correction.
Dreamer-SAC: Off-Policy Learning in Latent World Models for Sample-Efficient Autonomous Driving
This paper proposes Dreamer-SAC, a model-based reinforcement learning framework that integrates a recurrent state-space world model with soft actor-critic in latent space for sample-efficient autonomous driving. It outperforms DreamerV3, SAC, and PPO baselines while requiring fewer real environment interactions.
Stealing Reasoning Traces from Proprietary LLM APIs
A new paper reveals a vulnerability in proprietary LLM APIs where encrypted chain-of-thought blocks can be replayed across models and decrypted by jailbreaking weaker sibling models, exposing hidden reasoning traces. The issue has since been fixed by providers.