video-world-models

SolarWM introduces an open framework and unified training recipe for building interactive video world models with scalable training across diverse data sources, enabling long-horizon real-time rollouts.

ForgeWM is a progressive framework that distills bidirectional video generators into efficient few-step interactive world models, supporting low-latency interaction and replay-time refinement with demonstrated improvements on Minecraft and FPS gameplay.

A best-paper research from NVIDIA and collaborators introduces WorldTrace, a training-free framework that keeps compressed memory addressable in autoregressive video world models by assigning fixed slot-rank positions, enabling coherent long rollouts and long-range recall beyond the training horizon.

Introduces Latent Dynamics Reasoning (LDR), a video world model that integrates kinematic dynamics in a structured latent space, enabling extrapolation of learned dynamics far beyond training distributions while using far fewer parameters and running much faster than video diffusion baselines.

This paper introduces WorldTrace, a training-free memory framework for long-horizon video world models that keeps compressed cache addressable, plus LoopBench, a benchmark for episodic recall after long detours. It improves temporal consistency by +15.5% and episodic recall by +19.5% on LoopBench.

HelloWorld is a video world model that enables socially interactive characters, allowing users to prompt on-screen characters to respond via a single button press. It uses self-distillation and training-free cross-attention masking to naturalize interactions, and introduces HelloWorldBench for evaluation.

MiniWorld is a reproducible framework for training video world models from scratch using a block-causal Video Diffusion Transformer with Flow Matching, enabling efficient streaming generation and trainable in days on a single 8-GPU server.

ShadowDancer proposes a method for any-action, frame-level control of interactive video world models by learning unified dynamics representations from a video and its shadow, enabling transferable action control without labels or motion estimators. Experiments show improved action transfer and rollout performance over baselines.

AlayaWorld is an open-source framework for building interactive generative worlds that enables real-time user interaction and supports diverse actions. It unifies the complete development pipeline from data preparation to deployment.

MemLearner proposes a learning-based adaptive context query method using query tokens to improve scene consistency and memory in video world models, particularly for long sequences with occlusions and dynamic objects.

Holo-World presents a unified controllable video world model that generates videos from a single image with explicit control over camera, object motion, and weather. It introduces a novel dataset and techniques to preserve scene structure while transferring to target weather states.

This paper introduces MBench, a benchmark for evaluating the memory capabilities of video world models across entity, environment, and causal consistency over long temporal horizons.

This paper introduces latent spatial memory for video world models, storing 3D scene information directly in diffusion latent space to avoid costly pixel-space reconstruction. The proposed Mirage framework achieves up to 10.57x faster generation and 55x memory reduction while achieving state-of-the-art performance on WorldScore and RealEstate10K.

StressDream enhances video world models by steering diffusion-based imaginations toward high-impact yet plausible outcomes through optimized noise initialization with semantic and plausibility objectives, enabling robust policy evaluation and improvement.

minWM is a full-stack open-source framework that converts bidirectional video diffusion models into real-time interactive video world models with controllable camera, low-latency rollout, and modular architecture.

Incantation presents an interactive video world model that uses natural language as the action interface for fine-grained multi-entity control and cross-entity generalization, achieving high performance and real-time streaming through novel attention and distillation techniques.

MultiWorld is a unified framework for multi-agent multi-view video world modeling that achieves accurate control of multiple agents while maintaining multi-view consistency through a Multi-Agent Condition Module and Global State Encoder.