Paper page - Bernini: Latent Semantic Planning for Video Diffusion

Source: https://huggingface.co/papers/2605.22344 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

A unified video generation and editing framework combines multimodal large language models for semantic planning with diffusion models for pixel rendering, achieving state-of-the-art performance through semantic interface separation and enhanced positional embeddings.

Multimodal large language models(MLLMs) anddiffusion modelshave each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, whilediffusion modelssynthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs performsemantic planning, whilediffusion modelsrender pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework forvideo generationand editing. An MLLM-based planner predicts the target semantic representation directly in theViT embedding space, and aDiT-based renderersynthesizes pixels conditioned on this plan, augmented bytext featuresand, for editing, sourceVAE featuresfor detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduceSegment-Aware 3D Rotary Positional Embedding(SA-3D RoPE), and further incorporatechain-of-thought reasoningin the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range ofvideo generationand editing benchmarks, with the MLLM’s pretrained understanding translating into strong generalization on challenging editing tasks.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2605\.22344

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.22344 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.22344 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.22344 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.