The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

Hugging Face Daily Papers Papers
video-diffusion hand-motion-reconstruction 4d-pose computer-vision embodied-ai pretrained-models

Summary

ViDiHand leverages pretrained video diffusion model representations to reconstruct 4D hand motion directly from egocentric video frames, outperforming existing methods on ARCTIC, HOT3D, and HOI4D without detectors or optimization.

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.
Original Article
View Cached Full Text

Cached at: 06/30/26, 07:35 AM

Paper page - The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

Source: https://huggingface.co/papers/2606.30308

Abstract

ViDiHand uses pretrained video diffusion model representations with hand-overlay rendering to reconstruct 4D hand motion directly from video frames without detectors or optimization.

4D hand motion reconstructionfromegocentric videois bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely ontemporal moduleslearned only from scarcehand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly whatvideo generative modelsmust implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via ahand-overlay renderingobjective that specializes its features for hands while preserving its world priors. A decoder then recoversmetric-scale posefrom the adapted features. The whole pipeline operates directly onfull frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishingvideo diffusion modelsas a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2606\.30308

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.30308 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.30308 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.30308 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles