Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

Hugging Face Daily Papers Papers
world-model video-generation robotics multi-embodiment diffusion-transformer action-modeling vision

Summary

Dream4ACT introduces a shared visual action interface ("action views") rendered via URDF-based forward kinematics to enable joint video-action modeling across multiple robot embodiments, combining observations and actions in a single diffusion world model with masked flow matching. It achieves 88.98% success on RoboTwin 2.0 and 65.66 on TriWorldBench using a training-free URDF-constrained recovery mechanism for executable actions.

Video generation models (VGMs) offer strong spatiotemporal priors for embodied observation--action modeling. However, joint-space action vectors lack explicit image-space structure and vary in dimensionality and semantics across embodiments, making it challenging to directly leverage the rich spatiotemporal priors of VGMs. End-effector visualizations provide an alternative but do not specify the full articulated configuration needed for robot execution. We present Dream4ACT, a world model built for joint video-action modeling across embodiments. To unify action representations across embodiments, we introduce a shared visual action interface, called action views, which render target joint configurations from four prescribed virtual cameras using URDF-based forward kinematics. This shared visual representation preserves embodiment-specific articulated geometry while allowing observation and action sequences to share a video autoencoder and diffusion transformer. Through masked flow-matching, our model supports forward dynamics, inverse dynamics, and joint observation--action generation within a single jointly trained model by varying which future sequences are corrupted. To recover executable action sequences from predicted action views, we propose a training-free, URDF-constrained multiview recovery mechanism, without a learned embodiment-specific decoder. Dream4ACT achieves an average success rate of 88.98\% on RoboTwin~2.0 and an overall score of 65.66 on TriWorldBench, supporting effective closed-loop manipulation and competitive action-conditioned multiview prediction through the visual action interface.
Original Article
View Cached Full Text

Cached at: 10/05/26, 08:46 AM

Paper page - Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video-Action Modeling

Source: https://huggingface.co/papers/2609.40153

Abstract

Videogenerationmodels(VGMs)offerstrongspatiotemporalpriorsforembodiedobservation--actionmodeling.However,joint-spaceactionvectorslackexplicitimage-spacestructureandvaryindimensionalityandsemanticsacrossembodiments,makingitchallengingtodirectlyleveragetherichspatiotemporalpriorsofVGMs.End-effectorvisualizationsprovideanalternativebutdonotspecifythefullarticulatedconfigurationneededforrobotexecution.WepresentDream4ACT,aworldmodelbuiltforjointvideo-actionmodelingacrossembodiments.Tounifyactionrepresentationsacrossembodiments,weintroduceasharedvisualactioninterface,calledactionviews,whichrendertargetjointconfigurationsfromfourprescribedvirtualcamerasusingURDF-basedforwardkinematics.Thissharedvisualrepresentationpreservesembodiment-specificarticulatedgeometrywhileallowingobservationandactionsequencestoshareavideoautoencoderanddiffusiontransformer.Throughmaskedflow-matching,ourmodelsupportsforwarddynamics,inversedynamics,andjointobservation--actiongenerationwithinasinglejointlytrainedmodelbyvaryingwhichfuturesequencesarecorrupted.Torecoverexecutableactionsequencesfrompredictedactionviews,weproposeatraining-free,URDF-constrainedmultiviewrecoverymechanism,withoutalearnedembodiment-specificdecoder.Dream4ACTachievesanaveragesuccessrateof88.98\%onRoboTwin~2.0andanoverallscoreof65.66onTriWorldBench,supportingeffectiveclosed-loopmanipulationandcompetitiveaction-conditionedmultiviewpredictionthroughthevisualactioninterface.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.40153

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.40153 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.40153 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.40153 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Masked Visual Actions for Unified World Modeling

Hugging Face Daily Papers

Introduces Masked Visual Actions, a pixel-space control interface that expresses actions as partially revealed trajectories, enabling a single model to act as forward dynamics model, recover robot behavior, and support model-based planning and inverse modeling with only 15 hours of training data.

τ_0-WM: A Unified Video-Action World Model for Robotic Manipulation

Hugging Face Daily Papers

τ_0-WM is a unified video-action world model for robotic manipulation that integrates policy learning, video prediction, and action evaluation using a shared video diffusion backbone. It shows superior performance on challenging long-horizon and fine-grained tasks.