Masked Visual Actions for Unified World Modeling

Hugging Face Daily Papers Papers
world-modeling robotics visual-actions pixel-space-control forward-dynamics inverse-modeling policy-evaluation

Summary

Introduces Masked Visual Actions, a pixel-space control interface that expresses actions as partially revealed trajectories, enabling a single model to act as forward dynamics model, recover robot behavior, and support model-based planning and inverse modeling with only 15 hours of training data.

Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation. We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video. Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome. Finetuned with only 15 hours of masked examples from real videos and simulation, a single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. In downstream manipulation settings, the model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing robot motion from desired object motion.
Original Article
View Cached Full Text

Cached at: 07/22/26, 02:40 AM

Paper page - Masked Visual Actions for Unified World Modeling

Source: https://huggingface.co/papers/2607.19343 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Videomodelsabsorbrichpriorsoverhowthevisualworldmoves,interacts,andrespondstocontact,makingthempromisingsubstratesforroboticworldmodeling.Thecentralchallengeishowtocommunicateactiontosuchmodelsinaformalignedwiththevisualspaceinwhichtheylearnedtheseinteractionpriors,yetstillgroundedinphysicalmanipulation.WeintroduceMaskedVisualActions,apixel-spacecontrolinterfacethatexpressesactionasapartiallyrevealedtrajectoryofanarbitraryentityinavideo.Revealingrobotmotionmakesthemodelactasaforwarddynamicsmodelthatpredictsthescene’sresponsetolow-levelrobotactions,whilerevealingdesiredobjectmotionmakesthesamemodelrecoverrobotbehaviorconsistentwiththatoutcome.Finetunedwithonly15hoursofmaskedexamplesfromrealvideosandsimulation,asinglecheckpointachievesstrongvisualfidelityandcontrollabilityacrossdiversescenesandmultipleembodiments.Indownstreammanipulationsettings,themodelproducesimaginedrolloutswhoseoutcomescorrelatewithreal-worldexecutionforpolicyevaluation,improvesdecisionmakingbyrankingcandidatefuturesinmodel-basedplanning,andsupportsinversemodelingbysynthesizingrobotmotionfromdesiredobjectmotion.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.19343

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.19343 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.19343 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.19343 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

τ_0-WM: A Unified Video-Action World Model for Robotic Manipulation

Hugging Face Daily Papers

τ_0-WM is a unified video-action world model for robotic manipulation that integrates policy learning, video prediction, and action evaluation using a shared video diffusion backbone. It shows superior performance on challenging long-horizon and fine-grained tasks.

ABot-M0.5: Unified Mobility-and-Manipulation World Action Model

Hugging Face Daily Papers

ABot-M0.5 is a new World Action Model for mobile manipulation that improves performance through temporal granularity alignment, action space disentanglement, and train-test consistency, achieving state-of-the-art results on long-horizon and fine-grained manipulation benchmarks.