VideoMDM: Towards 3D Human Motion Generation From 2D Supervision
Summary
VideoMDM trains 3D human motion priors from 2D poses using a diffusion framework with 2D reprojection loss and 3D motion regularizers, achieving near-3D supervised performance without requiring 3D ground truth.
View Cached Full Text
Cached at: 06/12/26, 02:52 AM
Paper page - VideoMDM: Towards 3D Human Motion Generation From 2D Supervision
Source: https://huggingface.co/papers/2606.13364
Abstract
VideoMDM trains 3D human motion priors from 2D poses using a diffusion framework with 2D reprojection loss and 3D motion regularizers, achieving near-3D supervised performance without requiring 3D ground truth.
We introduce VideoMDM, adiffusion-based frameworkthat trains3D human motion priorsdirectly from accurate2D posesextracted frommonocular videos, without any 3D ground truth. A pretrained2D-to-3D lifterprovides approximate 3D pose sequences that serve as a noisy teacher: these are diffused, denoised by the model in 3D, and supervised in 2D by reprojecting the prediction and comparing against accurate keypoints. We show that, under mild assumptions, adepth-weighted 2D reprojection lossis equivalent in expectation to direct 3D supervision, and we adapt standard 3D motion regularizers -velocity consistencyandover-parameterized representation alignment- to this 2D setting. Unlike methods that lift 2D to 3D only at inference, VideoMDM learns a coherent3D motion manifoldduring training. OnHumanML3Dit nearly closes the gap to fully 3D-supervised MDM (FID 0.88 vs 0.54); On real video datasetsFit3DandNBAthe method learns to generate motions consistently preferred by humans, with strong quantitative results.
View arXiv pageView PDFProject pageGitHubAdd to collection
Get this paper in your agent:
hf papers read 2606\.13364
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.13364 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.13364 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.13364 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation
MVTrack4Gen introduces a training framework that uses multi-view point tracking as geometric supervision to enhance motion-aware diffusion models, achieving state-of-the-art geometric consistency and motion fidelity in novel-view video generation from monocular video.
The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction
ViDiHand leverages pretrained video diffusion model representations to reconstruct 4D hand motion directly from egocentric video frames, outperforming existing methods on ARCTIC, HOT3D, and HOI4D without detectors or optimization.
PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation
PhyMotion proposes a physics-grounded reward system that evaluates kinematic plausibility, contact consistency, and dynamic feasibility of human motion in generated videos, achieving stronger correlation with human judgment and improving motion realism in RL-based post-training.
DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents
DreamTraj predicts 6-DoF object trajectories from a single RGB image and a language instruction by decoding internal video diffusion latents, eliminating the need for video, depth, or CAD models at inference. It introduces the MOVEdataset with fine-grained language-to-motion annotations and achieves state-of-the-art performance while running 4.6x faster than generate-then-extract pipelines.
ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation
ARDY introduces a streaming generation framework for real-time, high-fidelity 3D human motion generation controlled by text and kinematic constraints, using a hybrid representation and two-stage autoregressive transformer denoiser.