GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Summary
GRAIL generates diverse humanoid manipulation and locomotion data using 3D assets and video foundation models, enabling effective sim-to-real transfer for humanoid robot control with high real-world success rates.
View Cached Full Text
Cached at: 06/04/26, 03:41 AM
Paper page - GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Source: https://huggingface.co/papers/2606.05160 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
GRAIL generates diverse humanoid manipulation and locomotion data through 3D asset composition and video foundation models, enabling effective sim-to-real transfer for robot control.
Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We present GRAIL, a digital generation pipeline that remains fully virtual until deployment: it composes 3D assets, simulator-ready scenes, and priors fromvideo foundation models(VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot. Rather than reconstructing unconstrained in-the-wild videos, GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction. This privileged setup better conditions4D recovery, allowing model-based object tracking, human motion estimation, and interaction-aware optimization to reconstruct metric 4Dhuman-object interaction(HOI) trajectories with reduced depth ambiguity and morphology mismatch. We retarget the recovered motions to ahumanoid robotand train complementary task-general trackers: anobject-aware latent adaptorfor manipulation and ascene-aware trackerfor terrain traversal. GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, we trainegocentric visual policiesthrough asim-to-real pipelineand deploy them on a Unitree G1 humanoid, achieving 84\% real-world success on diverse object pick-up and 90\% success on stair-climbing.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2606\.05160
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.05160 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.05160 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.05160 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation
OASIS is a simulation-data-driven framework for humanoid loco-manipulation that uses 3D generative models and hierarchical visuomotor policies. It achieves better zero-shot performance than real-robot training by leveraging domain randomization in simulation.
LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation
This paper introduces LUCID, a hierarchical model-based reinforcement learning framework for long-horizon humanoid loco-manipulation. It learns reusable latent skills and a macro-dynamics world model, enabling high-level planning via imagined rollouts and improving success rates in simulated multi-object rearrangement tasks.
DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation
DeVI introduces a framework that turns text-conditioned synthetic videos into physically plausible dexterous robot control via a hybrid 3D-2D tracking reward, enabling zero-shot generalization to unseen objects.
Human Universal Grasping
A flow-matching model generates diverse human grasps from RGB-D images, enabling zero-shot robotic grasping with improved performance over existing methods. The model, trained on a large egocentric dataset, significantly outperforms state-of-the-art baselines on a new benchmark.
Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data
Ego2Robot is a scalable pipeline that converts egocentric human manipulation videos into robot training data via action retargeting and visual synthesis, producing 18,561 hours of data across 15 robot morphologies. Experiments show that joint pretraining on this synthesized data improves out-of-distribution generalization for vision-language-action models, including on real-robot deployment.