CogOmniControl: Reasoning-Driven Controllable Video Generation via Creative Intent Cognition

Hugging Face Daily Papers Papers
video-generation diffusion-models reasoning controllable-generation creative-intent cogvlm reinforcement-learning

Summary

CogOmniControl is a reasoning-driven framework for controllable video generation that uses a specialized vision-language model (CogVLM) trained on anime production data to infer creative intent from sparse conditions, then guides a diffusion-based generator via reinforcement learning, achieving state-of-the-art results on new benchmarks.

Recent diffusion models achieve strong photorealism and fluency in video generation, yet remain fragile under abstract, sparse or complex conditions, leading to poor performance in professional production workflows such as storyboard sketches and clay render conditions. Existing video generation models, either inject conditions through adapters or couple a generic vision-language model (VLM) within a diffusion backbone, leaving a capability gap and failing to produce the videos that align with the user's creative intent. We present CogOmniControl, a reasoning-driven framework that factorizes controllable video generation into creative intent cognition and generation. Specifically, we train a specialized CogVLM using authentic anime production data. Compared to generic VLMs, it generates more professional and clear outputs, accurately cognizing user creative intent from sparse and abstract conditions and tuning these cues into dense reasoning output. Besides, CogOmniDiT unifies the controls from various conditions through in-context generation and is aligned to the CogVLM reasoning outputs via reinforcement learning. Furthermore, leveraging CogVLM's robust capability in guiding video generation, we release its potential in planning specific evaluators and enable a Best-of-N selection for the generated videos. This integration transforms the entire framework into a closed-loop "harness-like" architecture. We further introduce CogReasonBench and CogControlBench, built from professional workflows data that carry genuine creative intent rather than simulated ones. Experiments on two benchmarks show that CogOmniControl surpassed the existing open-source models. The project website: https://um-lab.github.io/CogOmniControl/
Original Article
View Cached Full Text

Cached at: 05/20/26, 06:37 AM

Paper page - CogOmniControl: Reasoning-Driven Controllable Video Generation via Creative Intent Cognition

Source: https://huggingface.co/papers/2605.19995

Abstract

Diffusion models applied in compressed image space generate high-quality images with lower computational cost and support flexible inputs like text or boxes.

Recentdiffusion modelsachieve strong photorealism and fluency invideo generation, yet remain fragile under abstract, sparse or complex conditions, leading to poor performance in professional production workflows such as storyboard sketches and clay render conditions. Existingvideo generationmodels, either inject conditions through adapters or couple a generic vision-language model (VLM) within a diffusion backbone, leaving a capability gap and failing to produce the videos that align with the user’s creative intent. We present CogOmniControl, a reasoning-driven framework that factorizes controllablevideo generationintocreative intent cognitionand generation. Specifically, we train a specializedCogVLMusing authentic anime production data. Compared to generic VLMs, it generates more professional and clear outputs, accurately cognizing user creative intent from sparse and abstract conditions and tuning these cues into dense reasoning output. Besides, CogOmniDiT unifies the controls from various conditions throughin-context generationand is aligned to theCogVLMreasoning outputs viareinforcement learning. Furthermore, leveragingCogVLM’s robust capability in guidingvideo generation, we release its potential in planning specific evaluators and enable aBest-of-N selectionfor the generated videos. This integration transforms the entire framework into a closed-loop “harness-like” architecture. We further introduceCogReasonBenchandCogControlBench, built from professional workflows data that carry genuine creative intent rather than simulated ones. Experiments on two benchmarks show that CogOmniControl surpassed the existing open-source models. The project website: https://um-lab.github.io/CogOmniControl/

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2605\.19995

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.19995 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.19995 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.19995 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MotiMotion: Motion-Controlled Video Generation with Visual Reasoning

Hugging Face Daily Papers

MotiMotion introduces a reasoning-then-generation framework for motion-controlled video generation that uses vision-language reasoning to refine trajectories and a confidence-aware control scheme to improve plausibility, outperforming existing approaches on a new benchmark.

OpenCoF: Learning to Reason Through Video Generation

Hugging Face Daily Papers

OpenCoF introduces a reasoning video dataset and a fine-tuned video generation model that improves temporal reasoning through diverse supervision and explicit reasoning tokens, showing significant gains on four video reasoning benchmarks.

Unlocking Complex Visual Generation via Closed-Loop Verified Reasoning

Hugging Face Daily Papers

Introduces CLVR (Closed-Loop Visual Reasoning), a framework that reformulates text-to-image generation from a single-step process into a closed-loop, multi-step visual reasoning approach using a VLM controller and diffusion models, achieving improved performance on compositional prompts.