Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
摘要
Introduces Latent Dynamics Reasoning (LDR), a video world model that integrates kinematic dynamics in a structured latent space, enabling extrapolation of learned dynamics far beyond training distributions while using far fewer parameters and running much faster than video diffusion baselines.
查看缓存全文
缓存时间: 2026/08/13 19:24
Paper page - Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
Source: https://huggingface.co/papers/2608.09926
Abstract
Latent Dynamics Reasoning integrates kinematic dynamics in structured latent space to enable video world models that extrapolate physical laws far beyond training distributions with far fewer parameters and faster inference.
The world evolves following its dynamics, i.e., its laws of motion. However, leadingvideo diffusionmodels largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduceLatent Dynamics Reasoning(LDR). LDR casts the latent transition as an explicitkinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on astructured latentrather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-boxphysics benchmarkspanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20times smaller than thevideo diffusionbaseline’s, under both single- and joint-task training at 256^2 resolution, while using 26times fewer parameters and running 143times faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first videoworld modelthat extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/
View arXiv pageView PDFProject pageGitHub28Add to collection
Get this paper in your agent:
hf papers read 2608\.09926
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### haodongli/LDR Updatedabout 1 hour ago • 8
Datasets citing this paper1
#### haodongli/LDR Updatedabout 1 hour ago • 115 • 7
Spaces citing this paper1
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
相似文章
超越像素:从视频先验到4D世界
本文介绍了Latent-to-4D,一种直接从视频扩散潜变量生成4D场景的方法,无需跨生成器重新训练,在几何和时间稳定性上优于级联方法。
使用潜空间世界模型预测后果并强化导航策略
本文提出了一种用于机器人导航的潜空间世界模型(LWM),该模型预测基于动作条件的潜特征兼容性,使得能够从无标签视频数据中学习策略,并通过无需额外环境交互的强化学习来提升性能,优于现有方法。
为何通用人工智能需要世界模型:大型语言模型的不足与世界模型的潜在优势
本文认为,大型语言模型在因果推理和长时域规划方面存在困难,其原因在于序列预测与对潜在环境动态的推理之间存在目标层面的不匹配,并引入了潜在动态推断视角以及Flux环境来研究这些局限性。
用于视频世界模型的潜在空间记忆
本文介绍了用于视频世界模型的潜在空间记忆,将3D场景信息直接存储在扩散潜在空间中,以避免昂贵的像素空间重建。所提出的Mirage框架实现了高达10.57倍的生成加速和55倍的内存缩减,同时在WorldScore和RealEstate10K上取得了最先进的性能。
通过残差潜在动作学习基于视觉特征的世界模型
本文介绍了 RLA-WM,一种基于视觉特征的世界模型,该模型利用残差潜在动作与流匹配技术高效预测未来视觉状态。该方法性能优于现有的视频扩散与特征基方法,同时支持从离线、无动作演示视频中探索新型机器人学习技术。