Video generation models as world simulators

# Video generation models as world simulators Source: [https://openai.com/index/video-generation-models-as-world-simulators/](https://openai.com/index/video-generation-models-as-world-simulators/) This technical report focuses on \(1\) our method for turning visual data of all types into a unified representation that enables large\-scale training of generative models, and \(2\) qualitative evaluation of Sora’s capabilities and limitations\. Model and implementation details are not included in this report\. Much prior work has studied generative modeling of video data using a variety of methods, including recurrent networks,[1](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-1),[2](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-2),[3](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-3)generative adversarial networks,[4](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-4),[5](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-5),[6](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-6),[7](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-7)autoregressive transformers,[8](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-8),[9](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-9)and diffusion models\.[10](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-10),[11](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-11),[12](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-12)These works often focus on a narrow category of visual data, on shorter videos, or on videos of a fixed size\. Sora is a generalist model of visual data—it can generate videos and images spanning diverse durations, aspect ratios and resolutions, up to a full minute of high definition video\. We take inspiration from large language models which acquire generalist capabilities by training on internet\-scale data\.[13](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-13),[14](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-14)The success of the LLM paradigm is enabled in part by the use of tokens that elegantly unify diverse modalities of text—code, math and various natural languages\. In this work, we consider how generative models of visual data can inherit such benefits\. Whereas LLMs have text tokens, Sora has visual*patches*\. Patches have previously been shown to be an effective representation for models of visual data\.[15](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-15),[16](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-16),[17](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-17),[18](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-18)We find that patches are a highly\-scalable and effective representation for training generative models on diverse types of videos and images\. Sora is a diffusion model[21](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-21),[22](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-22),[23](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-23),[24](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-24),[25](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-25); given input noisy patches \(and conditioning information like text prompts\), it’s trained to predict the original “clean” patches\. Importantly, Sora is a diffusion*transformer*\.[26](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-26)Transformers have demonstrated remarkable scaling properties across a variety of domains, including language modeling,[13](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-13),[14](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-14)computer vision,[15](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-15),[16](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-16),[17](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-17),[18](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-18)and image generation\.[27](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-27),[28](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-28),[29](https://openai.com/index/video-generation-models-as-world-simulators/#citation-bottom-29) **Long\-range coherence and object permanence\.**A significant challenge for video generation systems has been maintaining temporal consistency when sampling long videos\. We find that Sora is often, though not always, able to effectively model both short\- and long\-range dependencies\. For example, our model can persist people, animals and objects even when they are occluded or leave the frame\. Likewise, it can generate multiple shots of the same character in a single sample, maintaining their appearance throughout the video\.