PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

Hugging Face Daily Papers Papers
robotics dexterous-manipulation 3d-point-clouds world-action-models human-demonstration-pretraining vla

Summary

PointWAM introduces a 3D world action model that decomposes the world into scene and hands, jointly forecasting both as 3D point trajectories to enable dexterous robotic manipulation. Pre-training on human demonstration videos boosts DexJoCo success by 56.9 points, surpassing prior state-of-the-art by 11.7 points and outperforming strong VLAs on a real robot.

World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:53 AM

Paper page - PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

Source: https://huggingface.co/papers/2610.02840

Abstract

Worldactionmodelsjointlylearntoforecastworlddynamicsandpredictrobotactions,suchthatthelearnedinternalworlddynamicsguideaccurateactions.ExistingapproachestypicallyrepresenttheworldasRGBframesorlatentcounterpartswhilepredictingactionsasend-effectorposesorjointangles,buttheyoftenstruggletocapturethe3Dspatialstructureandcontactgeometrycentraltodexterousmanipulation.WeintroducePointWorldActionModel(PointWAM),a3Dworldactionmodelthatdecomposestheworldintoascene(i.e.,environment)andhands(i.e.,actor),andjointlyforecastsbothas3Dpointtrajectorieswithinasharedspace-timecoordinateframe.Thisexplicit,disentangledrepresentationenableseffectivepre-trainingonlarge-scalehumandemonstrationvideoswithoutrequiringanytask-specificobjectorkeypointselection.Givenacoloredpointcloudandalanguageinstruction,PointWAMpredictshowthesceneandhandsco-evolvein3Dspaceovertime,thenretargetstheforecasthandmotiontorobotactions.Pre-trainingonhumanvideosimprovesaverageDexJoCosuccessby56.9percentagepoints,andscene-trajectorysupervisionadds10.9pointsoverforecastingthehandsalone.Withboth,PointWAMsurpassesthepriorstateoftheartontenDexJoCotasksby11.7pointsandoutperformsstrongVLAsonarealrobot.

View arXiv pageView PDFProject pageAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.02840 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.02840 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.02840 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DeltaWAM: Delta World Action Models for Bimanual Manipulation

Hugging Face Daily Papers

DeltaWAM introduces delta-based world-action models for bimanual manipulation, enhancing efficiency and performance by predicting visual changes and actions. It demonstrates improved success rates and reduced computational overhead.

World Action Modeling with Progressive Visual Planning

arXiv cs.AI

ProWAM is a progressive world action model that jointly predicts robot actions and an ordered sequence of sparse visual sub-goals, enabling efficient long-horizon control without dense video generation. It achieves state-of-the-art results on LIBERO-Plus and RoboTwin, plus 70% zero-shot real-world success.

τ_0-WM: A Unified Video-Action World Model for Robotic Manipulation

Hugging Face Daily Papers

τ_0-WM is a unified video-action world model for robotic manipulation that integrates policy learning, video prediction, and action evaluation using a shared video diffusion backbone. It shows superior performance on challenging long-horizon and fine-grained tasks.