CogniRoute: Learning to Route Social Evidence in Omni-Modal Models
Summary
CogniRoute is a schema-guided Mixture-of-Experts framework for social video question answering that improves multimodal reasoning through cognitive schema factorization and route-aware reinforcement learning. It achieves significant gains over baselines on the new OmniSocialBench benchmark.
View Cached Full Text
Cached at: 06/29/26, 10:05 PM
Paper page - CogniRoute: Learning to Route Social Evidence in Omni-Modal Models
Source: https://huggingface.co/papers/2606.20970 Authors:
,
,
,
,
,
,
,
,
,
Abstract
CogniRoute is a schema-guided Mixture-of-Experts framework for social video question answering that improves multimodal reasoning through cognitive schema factorization and route-aware reinforcement learning.
Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, aschema-guidedMixture-of-Expertsframework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example bycross-modal relation,reasoning demand, andtemporal scope, and alignsglobal routing signatureswith this structure duringsupervised fine-tuning. We further introduceroute-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards foranswer correctness,modality-consistent reasoning, andcognitive temporal grounding. To support training and evaluation, we constructOmniSocialBench, a diagnostic social video QA resource with 118Kstructured training examples,grounded reasoning traces, schema labels,temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38\% average accuracy onOmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.20970 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.20970 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.20970 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes
AffectOmni is a GRPO-trained framework for verifiable affective reasoning in multimodal large language models, introducing People Focus and Temporal Order rewards to enhance people-centric evidence selection and temporally structured reasoning, with experiments showing improvements over 7B scale baselines.
Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
This paper introduces the Expert Subspace Separation Index (ESSI) to disentangle route coherence, candidate quality, and candidate-by-context interaction in sparse Mixture-of-Experts language models, revealing a pattern of coherent overlap where routing selects token-relevant experts from a shared geometric neighborhood yet multi-expert computation remains useful.
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
This paper proposes Multi-level Context Fusion MOE (MCF-MOE), a framework that improves routing consistency in Mixture-of-Experts models by integrating cross-layer semantic aggregation and local token-level interactions, outperforming strong baselines on language modeling and understanding benchmarks.
CORA: Analyzing and bridging thinking-answer gap in Multimodal RLVR via Consistency-Oriented Reasoning Alignment
This paper analyzes the thinking-answer inconsistency in multimodal reinforcement learning with verifiable rewards (RLVR) for large vision-language models and proposes CORA, a method that introduces a consistency reward model and hybrid reward advantage splitting to improve faithfulness and task performance.
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment
This paper proposes SARA, a framework that aligns routing distributions of multilingual inputs using Jensen-Shannon divergence to improve expert sharing for low-resource languages in sparse Mixture-of-Experts models. Experiments on Qwen3-30B-A3B and Phi-3.5-MoE-instruct show improvements on multilingual benchmarks.