Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning

Hugging Face Daily Papers 论文

摘要

Introduces Latent Dynamics Reasoning (LDR), a video world model that integrates kinematic dynamics in a structured latent space, enabling extrapolation of learned dynamics far beyond training distributions while using far fewer parameters and running much faster than video diffusion baselines.

The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20times smaller than the video diffusion baseline's, under both single- and joint-task training at 256^2 resolution, while using 26times fewer parameters and running 143times faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/
查看原文
查看缓存全文

缓存时间: 2026/08/13 19:24

Paper page - Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning

Source: https://huggingface.co/papers/2608.09926

Abstract

Latent Dynamics Reasoning integrates kinematic dynamics in structured latent space to enable video world models that extrapolate physical laws far beyond training distributions with far fewer parameters and faster inference.

The world evolves following its dynamics, i.e., its laws of motion. However, leadingvideo diffusionmodels largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduceLatent Dynamics Reasoning(LDR). LDR casts the latent transition as an explicitkinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on astructured latentrather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-boxphysics benchmarkspanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20times smaller than thevideo diffusionbaseline’s, under both single- and joint-task training at 256^2 resolution, while using 26times fewer parameters and running 143times faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first videoworld modelthat extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/

View arXiv pageView PDFProject pageGitHub28Add to collection

Get this paper in your agent:

hf papers read 2608\.09926

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### haodongli/LDR Updatedabout 1 hour ago • 8

Datasets citing this paper1

#### haodongli/LDR Updatedabout 1 hour ago • 115 • 7

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

相似文章

超越像素:从视频先验到4D世界

Hugging Face Daily Papers

本文介绍了Latent-to-4D,一种直接从视频扩散潜变量生成4D场景的方法,无需跨生成器重新训练,在几何和时间稳定性上优于级联方法。

使用潜空间世界模型预测后果并强化导航策略

arXiv cs.AI

本文提出了一种用于机器人导航的潜空间世界模型(LWM),该模型预测基于动作条件的潜特征兼容性,使得能够从无标签视频数据中学习策略,并通过无需额外环境交互的强化学习来提升性能,优于现有方法。

用于视频世界模型的潜在空间记忆

Hugging Face Daily Papers

本文介绍了用于视频世界模型的潜在空间记忆,将3D场景信息直接存储在扩散潜在空间中,以避免昂贵的像素空间重建。所提出的Mirage框架实现了高达10.57倍的生成加速和55倍的内存缩减,同时在WorldScore和RealEstate10K上取得了最先进的性能。

通过残差潜在动作学习基于视觉特征的世界模型

Hugging Face Daily Papers

本文介绍了 RLA-WM,一种基于视觉特征的世界模型,该模型利用残差潜在动作与流匹配技术高效预测未来视觉状态。该方法性能优于现有的视频扩散与特征基方法,同时支持从离线、无动作演示视频中探索新型机器人学习技术。