Robot actions can be represented as motion in pixel space.

NVIDIA Research introduces Hydra-0, a generalist world model conditioned on action flow: image-plane trajectories that enable one model to learn across human hands, handheld grippers, single-arm robots and bimanual systems.

Explore the project https://nvda.ws/4xaO9C0

Hydra-0: Action Flow for Generalist World Modeling and Control

Source: https://nvidia-isaac.github.io/video_to_data/hydra-0/?linkId=100000437297790

Overview video

**Figure 1: Action flow as a shared control interface.***Top:*Hydra-0 learns from diverse interaction videos featuring egocentric human demonstrations, handheld UMI grippers, bimanual robot arms, and single-arm robots.*Middle:*Visible embodiment motion is represented as image-plane flow trajectories, placing heterogeneous interactions in a common, pixel-aligned action flow space independent of embodiment-specific joint or end-effector coordinates. This unified interface enables a single generalist world model to learn from multi-embodiment data and transfer across interaction settings.*Bottom:*In forward mode, provided gripper flow conditions future scene prediction, whose rollouts support open-loop policy evaluation; in inverse mode, desired object flow elicits compatible robot motion that a supervised readout converts into executable actions.

Action flow as a shared visual interface

Figure 1, bands 1–2 · animated

Sources

Shared action flow

EgoDex · human hands

Deform360 · handheld gripper

DROID · single arm

XVLA-Soft-Fold · bimanual

One representation, four embodiments: an egocentric human hand, a handheld gripper, a single arm and a bimanual robot. The lower row is the upper row plus the interface — the same window carrying the trajectories the model is conditioned on. What that interface then buys is the rest of this page: simulation, policy evaluation, and control.

Abstract

We introduceHydra-0, a generalist world model conditioned onaction flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r = 0.96 between replayed and reference success rates.

Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.

90.4%

lower robot-motion error vs. native-action baseline

60.2%

lower object-motion error

0.96

correlation with RoboLab success over 300 episodes

16.0×

generation-only speedup after few-step distillation

2,202 h

filtered multi-embodiment training video

Method

One conditioning interface, built two ways: from video during training, from kinematics at deployment.

Embodiment — the acting body

Manipulated object

Action flow is a set of camera-plane trajectories describing the commanded motion of the visible embodiment: image positions with visibility flags, tracked over the prediction horizon. The same representation can describe a robot arm, a handheld gripper, or a human hand, without exposing any of their native action spaces to the video model.

Offline trainingaction flow recovered from interaction video

DROID single arm

Interaction video

Action flow

Deform360 handheld gripper

Interaction video

Action flow

XVLA-Soft-Fold bimanual arms

Interaction video

Action flow

Each pair is one five-second window played twice, frame-locked: on the right, the same window carrying the trajectories the model is conditioned on, drawn over a one-second rolling history. The row scrolls sideways. Tracks are annotated on both the embodiment and the manipulated object: a dense tracker recovers the trajectories from the video, and grounded masks split them. Nothing in this lane is projected from robot geometry — that is the deployment route below. The first frame plus this flow is the entire condition; the rest of the clip is the flow-matching target.

Online deploymenta hybrid loop: physics moves the robot, the world model moves the world

Deployment is not one simulator but two, and each does only what it is good at.Isaac Labowns the robot: a rigid, calibrated, fully known body whose response to a command is a solved problem, so it is executed exactly rather than predicted.Hydra-0owns everything the physics engine would need an asset and a material model for — the cloth, the deformable, the contact, the lighting, the scene it has never been given a mesh of. Action flow is the seam between them: it is what the physics enginehands over, and the only thing about the robot the video model is ever told.

1Observation the single frame the model is conditioned on2Command rollout in Isaac Lab the candidate command, executed by a physics simulator3Kinematically projected flow the hand-off — the same command, in the image plane4Predicted consequence what the world model generates Visible robot-surface points are propagated by the commanded link transforms and projected through the calibrated camera; the model then predicts the command’s consequence. Step 3 opens on the Isaac Lab render carrying its own projected flow and dissolves to the real observation carrying the same flow — both are the same calibrated top camera, which is why the tracks sit on the arms in either, and is the whole reason a command executed in simulation can condition a prediction about the real scene. The three clips are one seventeen-second closed-loop rollout on a single clock, so the arm pose in the simulator, the flow drawn on the observation and the generated frame all belong to the same instant.

Geometry-aware construction

When robot geometry and calibration are available, surface points visible in the first observation are propagated under the candidate command and projected into the camera plane. A point counts as visible only when it has positive camera depth, lies inside the image, and agrees with the rendered depth buffer within 1.2 cm over a 3×3 neighborhood, which is what rejects self-occluded points.

Video-only construction

Most large interaction datasets ship neither robot description files nor calibration. There we recover dense image-plane trajectories with a flow tracker and segment them with grounded embodiment and object masks — the same representation, no privileged metadata.

Grounding and sampling

Figure 2 · dense tracks are split by mask, then one conditioning mode is drawn per training step.

Embodiment — the acting body

Manipulated object

Dense tracks tracker output, before anything is labelledEmbodiment mask held on both arms as they moveManipulated-object mask held on the cloth as it deformsEmbodiment· p = 0.40 Acting-body tracks — the primary action condition.Object· p = 0.40 Task or desired-motion flow; the mode that inverts the model.All· p = 0.15 Fallback when semantic grounding is incomplete.None· p = 0.05 Conditioning dropout — text and image only.All seven panels are the same 81-frame window, played together. One conditioning mode is drawn per training step, from the pools the masks define: 1,819 embodiment tracks and 3,250 object tracks out of 14,997 the tracker returns.

Qualitative rollouts

Figure 3

Full five-second rollouts, one held-out dataset per row. Every panel in a row is the same evaluation sample at the same timestep, so a method reads down a column and a dataset reads across. Each row carries three held-out samples — step through them with the arrows beside the dataset name, and hover the counter to see the task the model was conditioned on.

Wan-Move

Cosmos 2.5

Ours

Ground truth

XVLA-Soft-Fold

bimanual YAM · garment folding

1 / 3

Deform360

handheld gripper · single-object manipulation

1 / 3

DROID

single Franka arm · in-the-wild scenes

1 / 3

ABC-130k

bimanual YAM · household chores

1 / 3

The four panels of a row play as one clip: the page pins them to a shared clock and corrects them on the frame a loop wraps, so a difference on screen is a difference between methods rather than between players. Each baseline is resampled into that sample’s ground-truth frame shape — Cosmos 2.5 renders every sample at 320×256, and Wan-Move at its own 720×544 to 832×464, against ground truths running 640×480 to 848×480 — so how much of a panel an object fills is not an artifact of who rendered larger.

Evaluation on multi-embodiment datasets

All methods see the same 100 validation clips per dataset across five held-out sets.

Within the same Cosmos 2.5 backbone, swapping the native relative 6D action for action flow improves PSNR, SSIM, gripper EPE, FID and FVD on all five datasets, and object EPE on the four where it is measured — a controlled comparison that isolates the conditioning representation. VLM scores are the exception, and are mixed: the judge rates physical plausibility and temporal consistency rather than whether the generated motion follows the commanded trajectories, which flow EPE measures directly. The best configuration is the distilled four-step Wan2.2 A14B model, best in nearly every cell and also the fastest.

ModelPSNR ↑SSIM ↑Obj. EPE ↓Grip. EPE ↓FID ↓FVD ↓VLM ↑

{{ row.name }}

{{ cell.text }}

Gray rows are zero-shot baselines using released checkpoints; the best value per column is bold green. Standard errors are reported in the paper. Object EPE is omitted for DROID, where object grounding is unreliable in cluttered scenes.

Wrist-camera egomotion

A camera bolted to the arm produces global image motion that a fixed viewpoint never sees. Given depth and relative camera pose, camera egomotion converts into the same image-space condition, so one interface covers both camera and interaction motion. This is a single-rollout proof of concept, not a systematic evaluation.

One DROID wrist window: ground truth, our generation, and the conditioning flow it was given.

Data efficiency from multi-embodiment mid-training

Six held-out Interactive World Simulator tasks, adapted with 0–100% of each task’s training split.

Data efficiency on the six held-out IWS tasks; lower is better on all three metrics.**Ours (MT)**starts from the multi-embodiment mid-trained checkpoint,**Ours (PT)**from the original Wan2.2 weights; Wan-Move and ATI are evaluated zero-shot.Strongest transfer before any task data

At 0%, mid-trained beats pretrained-only on LPIPS, object-flow EPE, and FVD across all six tasks. The task-specific IWS model and the newly added Cosmos 2.5 action layers are substantially worse on all three, so their unadapted action interfaces do not transfer to held-out tasks.

The lead survives adaptation

At 100%, mid-trained has the lowest LPIPS and FVD on all six tasks and the lowest flow EPE on four.

Most gains arrive by 20%

Between 20% and 100% of the task split, per-task values move at most 3.4% in LPIPS, 6.7% in flow EPE, and 6.8% in FVD.

Policy evaluation by open-loop replay

The simulator is initialized from an episode’s first observation and replays that episode’s achieved trajectory as action flow.

Because the policy is never queried on generated observations, this measures whether the simulator preserves an outcome it is shown — not prospective evaluation of an unexecuted command. Generated rollouts are scored by a human rater applying the same task predicate, with policy identity withheld.

RoboLab

5 policies × 6 tasks · 300 episodes

π₀, π₀.₅, GR00T N1.7, Cosmos-3 Nano, and Cosmos-3 Edge, rolled out 10 times per task.

Each point aggregates the 10 rollouts of one policy–task pair; the dashed line is the least-squares fit.- Aggregate success 26.3% simulated vs. 26.7% in RoboLab — a bias of −0.3 points.

• The two agree on the ordering of all five policies.
• Per-episode verdicts match 93.0% of the time (Cohen’s κ = 0.82): 10 false positives, 11 false negatives.

Side-by-side replay

reference vs. simulated, same reset and camera

RoboLab reference · successSimulated · successRoboLab reference · failureSimulated · failure

Running the interface backwards

Figure 4 · flexible-pipe bending, from the human demonstration through to the robot’s execution.

Supply desired object flow instead of embodiment flow and the world model generates a compatible robot motion, from which a learned action readout produces executable 14-DoF commands. The object flow here is transferred from a held-out human demonstration, and the readout is trained on paired real-world rollouts of both successes and failures — no task-specific expert robot demonstration is required.

Human demonstration held-out sourceDesired object flow the only condition suppliedGenerated robot motion no embodiment flow inputPhysical execution 14-DoF YAM, action readoutAll four clips come from one flexible-pipe run.

Limitations

The world action model can exhibit roughly 1 cm of grasp imprecision. We hypothesize that limited depth awareness contributes, but do not establish it as the cause. Grasp and contact state can also be ambiguous in generated rollouts, including whether an object has actually been secured. Conditioning on depth, tactile, or force signals is the obvious next step. The wrist-camera result is a qualitative DROID proof of concept; systematic evaluation under mobile manipulation and broader camera motion remains future work, as does closed-loop policy evaluation — everything reported here is open-loop.

BibTeX

If you find this work useful, please cite the report.

arXiv:2608.18077

@misc{li_hydra-0_2026,
    title = {Hydra-0: {Action} {Flow} for {Generalist} {World} {Modeling} and {Control}},
    shorttitle = {Hydra-0},
    url = {http://arxiv.org/abs/2608.18077},
    doi = {10.48550/arXiv.2608.18077},
    publisher = {arXiv},
    author = {Li, Hongyu and Wen, Bowen and Zhu, Xinghao and Wang, Yixuan and Du, Yilun and Li, Yunzhu and Konidaris, George and Birchfield, Stan and Pouya, Soha and Li, Chenran and Chang, Yan},
    month = aug,
    year = {2026},
    note = {arXiv:2608.18077 [cs.RO]},
}