World Models Explained: What Every AI Is Missing

TL;DR: World models are a key component for AI to simulate reality and predict the future, filling the gap left by LLMs that only learn from text. They enable training in virtual environments through imagination, driving breakthroughs in robotics and autonomous driving. ## What is a World Model? Imagine someone pushes a cup off a table—you reach to catch it before it hits the ground, because your brain has already simulated the whole process in advance. A toddler, never burned, will still pull their hand away from a fire. Everyone has a built-in "reality simulator" constantly predicting what happens next. This simulator is precisely what nearly all AI systems are missing today, and building it has become the biggest race in AI. Its name is the **world model**. A world model is a system that learns to predict what will happen next in an environment. Given a current state and an action you intend to take, it outputs the next state. Feed that output back into itself, and it can roll forward step by step, like a simulator that learns by watching. ## Predicting Words vs. Predicting the World Large language models (LLMs) predict the next word; world models predict the next moment. The difference is subtle, but it changes everything. Once a machine can imagine consequences before acting, it can plan ahead, practice, and learn from futures that never actually happen. Current LLMs can describe gravity and explain why eggs break, but they learn from text—from how humans talk about the world, not the world itself. Ask a top model to control a robotic hand to pour a glass of water, and it has no reliable sense of weight, balance, or what happens when the cup tilts. To act, a system needs an internal model of physical space, and that's exactly the gap world models fill. ## Core Blueprint: Vision + Memory + Controller The main blueprint for building these models came in 2018 from David Ha and Jürgen Schmidhuber. They split an agent into three parts: - **Vision**: Takes the full game screen and compresses it into a short list of numbers (latent space), preserving meaning while discarding pixels. - **Memory**: Takes the compressed scene plus the action, predicts the compressed version of the next moment, and learns the dynamics. - **Small controller**: Observes the current state and the predicted future to decide what to do. Once the memory network can predict the next moment, it no longer needs the real game—it can generate the entire environment itself: frame by frame, producing hallucinated trajectories inside the model. Researchers trained an agent to drive in this dream, never touching the real game, then deployed the learned policy back into reality—and it worked. Real experience is slow, expensive, and sometimes dangerous, but dream experience is nearly free—you can run millions of years in a single night. ## Holes in the Dream: Uncertainty is Key But there's a problem with the dream world: if the agent becomes too clever, it can "cheat" the dream, exploiting holes in the model (e.g., where enemies forget to shoot) to win, then crash in reality. The solution is almost philosophical: the world model must maintain uncertainty, inject noise into its predictions, and admit it doesn't know exactly what will happen next. An honestly uncertain model is hard to fool. ## Latent Space and Compression The model doesn't store images, but a short string of numbers—the latent space. This is a compressed summary of all important information at the moment. The model learns to roll this summary forward in time. Modern versions deliberately split this state into two parts: one deterministic (the part the model is sure about, passed forward like memory), and one stochastic (controlled randomness, representing everything truly unpredictable). Keeping a continuously updated compressed state allows the model to imagine thousands of steps cheaply—because it doesn't need to draw a single frame while thinking. ## The Dreamer Series: Training in Imagination Scaling this idea gives us a family of models called **Dreamer**. The latest, Dreamer v3, learns a compact model of the environment from its own experience, then trains its entire policy inside rollouts of this model's imagination. Its most famous achievement is collecting diamonds in *Minecraft*: chopping wood, crafting tools, digging through layers of stone, surviving with almost no reward until the end. Dreamer v3 was the first agent to complete this task entirely from scratch, without any human demonstration. And the same algorithm, with the same settings, handles hundreds of other tasks—from balancing a robot to playing Atari games. ## Two Camps: Pixels vs. Meaning At this point, the world model field splits into two camps: - **Pixel camp**: believes the model should predict the world's pixels in full, generating the exact next frame you would see (like a video). - **Meaning camp**: argues this is wasteful—the model only needs to predict the "meaning" of the next moment, using a compressed internal language, without the effort of drawing. ### Pixel Camp: GameNGen and Genie In 2024, Google researchers built **GameNGen**, which runs the classic shooter *Doom* without an underlying engine. No code tracks health, monsters, or the map; a single neural network watches the last few frames and buttons, drawing the next frame 20 times per second. It learned by watching an AI play thousands of hours of real *Doom*, mastering the rules directly from the video. The whole game lives inside the network's guess of "what should happen next." Then came Google DeepMind's **Genie** series, which can do the same from a single sentence. Input "a small village under a snowy mountain, with a dome," and Genie generates a world you can walk through in real time—72 pixels, 24 frames per second, consistent for minutes. DeepMind puts learning agents into these generated worlds for training, because producing endlessly diverse environments from text gives you an infinite gym to teach agents how to act. But this approach has a ceiling: the model is great at mimicking how the world *looks*, but far worse at how it *actually works*. Walk too far, and geometry drifts, corridors rearrange, objects vanish. That's fine for demo clips, but for a robot that must trust its predictions, the gap between "looks right" and "is right" is everything. Generating every pixel is extremely expensive computationally, with most effort wasted on details irrelevant to decision-making. ### Meaning Camp: JEPA and V-JEPA This is the entire argument of the second camp, most loudly voiced by **Yann LeCun**. His view: predicting every pixel forces the model to waste vast effort on things that are unpredictable and unimportant (e.g., the exact sway of leaves, static noise on a dead channel). So his architecture, **JEPA** (Joint Embedding Predictive Architecture), predicts the future only in representation space, deliberately discarding unpredictable details while retaining the meaningful parts. Meta's **V-JEPA 2** learned this way from over a million hours of video, then with only about 60 hours of robot data, it could plan real robotic actions in places it had never seen. It doesn't need to draw the future—only understand it enough to act. ## Hardware Companies Step In: NVIDIA Cosmos and Physical AI This is no longer just research. The clearest sign is what hardware companies are doing with it. NVIDIA built a family of world models called **Cosmos**, trained on about 20 million hours of driving, robotics, and real-world video. These models' job is to "dream up" training data: robots learn to grasp and walk inside these simulations before ever touching a real object. The industry has even coined a name for the whole frontier: **Physical AI**, and world models are how you train it at scale without breaking anything real. ## The Long Tail Problem in Autonomous Driving Autonomous driving is the most concrete example. 99% of driving is boring and easy—cars mastered it years ago. What still fails them is the one-in-a-million long-tail event: a sofa falling off a truck, a deer appearing at the edge of the headlights, a child chasing a ball into the road. You can't collect enough such events by driving more miles, and you absolutely cannot rehearse dangerous scenarios on real roads. So world models generate them: take the few rare fragments you have, turn them into thousands of variants in fog, at night, in rain, from new angles. The car can practice its most critical moments inside a simulator that will never send someone to the hospital. ## Spatial Intelligence: World Labs and Marble Another flavor worth knowing comes from **World Labs**, founded by **Fei-Fei Li**. Her concept is "spatial intelligence." Her team's model, **Marble**, can take a sentence or a single image and build a true 3D world—real geometry that you can fly through from any angle. The world stays solid even as you move. Her argument: language captures how we think, and space is the other half—the fundamental architecture underlying the behavior of any animal or robot. ## How to Try It Yourself - **Generators**: World Labs' Marble and DeepMind's Genie both run in the browser, no code required. - **Look inside**: Dreamer v3 and NVIDIA Cosmos models are open-source—download weights from Hugging Face. - **Train one**: The Dreamer code can learn to balance a pole or play an Atari game on a single GPU in one afternoon. If you want hands-on practice with AI/ML, find over 50 interactive labs on preparato.com/labs that run on real GPUs, helping you learn building agents, running RAG pipelines, fine-tuning small language models, and more. Source: https://youtu.be/BylqLPQvr1k?is=7Sg65etpKCtH2Aga