RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

Hugging Face Daily Papers Papers
cross-embodiment world-model robotics video-prediction latent-actions imitation-learning

Summary

RynnWorld-Latent is a cross-embodiment video world model that uses 3D-grounded latent actions as a shared conditioning interface, enabling joint pretraining on human videos, robot demonstrations, and simulation rollouts to generate synthetic data that improves robot policy performance on LIBERO and real-world platforms.

Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
Original Article
View Cached Full Text

Cached at: 10/28/26, 09:26 AM

Paper page - RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

Source: https://huggingface.co/papers/2610.36791 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

RynnWorld-Latent learns cross-embodiment world dynamics via 3D-grounded latent actions, enabling joint pretraining on diverse video sources and synthetic data generation for improved robot policy performance.

Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, across-embodiment video world modelthat uses the3D-grounded latent actionsintroduced byRynnLAMas a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrainedvideo transformer, RynnWorld-Latent combinesglobal action tokenswith anidentity-initialized local pathwaythat aligns actions with futurevideo latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes:digital teleoperationuses target robot commands to generate action-video demonstrations, whilelatent-action synthesistransfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates onLIBEROandLIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results supportuniversal latent actionsas a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.

View arXiv pageView PDFProject pageGitHub5Add to collection

Get this paper in your agent:

hf papers read 2610\.36791

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.36791 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.36791 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.36791 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Hugging Face Daily Papers

RynnWorld-4D is a generative world model that co-produces future RGB, depth, and optical flow from a single RGB-D image and language instruction using a unified diffusion process, enabling efficient robotic manipulation through inverse dynamics policy learning. It achieves state-of-the-art on real-world bimanual manipulation tasks.