DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

Hugging Face Daily Papers Papers
action-conditioned-video-world-model robotic-manipulation geometric-attention distillation depth-estimation world-model

Summary

DreamX-Phi 1.0 is an action-conditioned video world model for robotic manipulation that predicts future observations using geometric attention encoding, depth estimation, and distillation, achieving top results in the WorldArena 2.0 Challenge.

We present DreamX-Phi 1.0, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm SE(3) transformations into attention via PRoPE-style geometric encoding, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight depth branch for scene-level geometry and use SAM3 masks with a frozen V-JEPA teacher to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.
Original Article
View Cached Full Text

Cached at: 08/14/26, 07:27 AM

Paper page - DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

Source: https://huggingface.co/papers/2608.13489

Abstract

DreamX-Phi 1.0 is an action-conditioned video world model for robotic manipulation that uses geometric attention encoding, depth estimation, object masks with a frozen teacher, and distillation to generate faithful future observations.

We present DreamX-Phi 1.0, anaction-conditioned video world modelfor robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm’s commanded path, we inject per-armSE(3) transformationsinto attention viaPRoPE-style geometric encoding, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweightdepth branchfor scene-level geometry and useSAM3 maskswith a frozenV-JEPA teacherto maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student viadistribution-matching distillationfor efficient deployment. At the time of writing, achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.

View arXiv pageView PDFGitHub23Add to collection

Get this paper in your agent:

hf papers read 2608\.13489

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.13489 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.13489 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.13489 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DreamX-World 1.0: A General-Purpose Interactive World Model

Hugging Face Daily Papers

DreamX-World 1.0 is a general-purpose interactive text/image-to-video world model that supports camera navigation, scene persistence, and promptable events across multiple domains, using novel techniques like E-PRoPE, causal forcing, and memory-conditioned scene persistence to achieve controllable long-horizon generation.

τ_0-WM: A Unified Video-Action World Model for Robotic Manipulation

Hugging Face Daily Papers

τ_0-WM is a unified video-action world model for robotic manipulation that integrates policy learning, video prediction, and action evaluation using a shared video diffusion backbone. It shows superior performance on challenging long-horizon and fine-grained tasks.

Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

Hugging Face Daily Papers

Dream4ACT introduces a shared visual action interface ("action views") rendered via URDF-based forward kinematics to enable joint video-action modeling across multiple robot embodiments, combining observations and actions in a single diffusion world model with masked flow matching. It achieves 88.98% success on RoboTwin 2.0 and 65.66 on TriWorldBench using a training-free URDF-constrained recovery mechanism for executable actions.

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

Hugging Face Daily Papers

PointWAM introduces a 3D world action model that decomposes the world into scene and hands, jointly forecasting both as 3D point trajectories to enable dexterous robotic manipulation. Pre-training on human demonstration videos boosts DexJoCo success by 56.9 points, surpassing prior state-of-the-art by 11.7 points and outperforming strong VLAs on a real robot.