HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesis

Hugging Face Daily Papers Papers
hand-object-interaction multi-view-synthesis diffusion-model 3d-motion video-generation

Summary

HarmoHOI is a unified diffusion framework that jointly generates synchronized multi-view hand-object interaction videos and globally aligned 3D point tracks, achieving state-of-the-art performance in visual quality, motion plausibility, and multi-view geometric consistency.

Hand-Object Interaction (HOI) synthesis is a cornerstone for animation production and embodied AI. Despite the strong priors of video foundation models, multi-view consistent HOI synthesis remains challenging due to complex hand motions and occlusions. We present HarmoHOI, a unified diffusion framework that jointly and harmoniously generates synchronized multi-view HOI videos and globally aligned 3D point tracks. Our core insight is that robust multi-view consistency fundamentally requires globally aligned 3D geometry and motion. To this end, we propose a Mixture of Multi-view Diffusion Transformer that co-models RGB videos and 3D point tracks. By representing point tracks as pseudo-videos, we align 3D geometric signals with the 2D latent space of foundation models, thereby minimizing the domain gap and easing adaptation of priors. To further ensure geometry consistency, we introduce Global Motion Aligning Diffusion, which refines coarse point tracks into metric-scale, globally aligned 3D trajectories. HarmoHOI enables on-the-fly co-evolution of 2D appearance and 3D motion during denoising. To overcome the scarcity of multi-view HOI data, we employ a hybrid data curriculum learning strategy that successfully transfers generic priors from single-view data to synchronized multi-view generation. Experimental results show that HarmoHOI achieves state-of-the-art performance in visual quality, motion plausibility, and multi-view geometric consistency. Project page available at https://droliven.github.io/HarmoHOI_project.
Original Article
View Cached Full Text

Cached at: 07/21/26, 10:36 AM

Paper page - HarmoHOI: Harmonizing Appearance and 3D Motion for Multi-view Hand-Object Interaction Synthesis

Source: https://huggingface.co/papers/2607.17097

Abstract

Hand-ObjectInteraction(HOI)synthesisisacornerstoneforanimationproductionandembodiedAI.Despitethestrongpriorsofvideofoundationmodels,multi-viewconsistentHOIsynthesisremainschallengingduetocomplexhandmotionsandocclusions.WepresentHarmoHOI,aunifieddiffusionframeworkthatjointlyandharmoniouslygeneratessynchronizedmulti-viewHOIvideosandgloballyaligned3Dpointtracks.Ourcoreinsightisthatrobustmulti-viewconsistencyfundamentallyrequiresgloballyaligned3Dgeometryandmotion.Tothisend,weproposeaMixtureofMulti-viewDiffusionTransformerthatco-modelsRGBvideosand3Dpointtracks.Byrepresentingpointtracksaspseudo-videos,wealign3Dgeometricsignalswiththe2Dlatentspaceoffoundationmodels,therebyminimizingthedomaingapandeasingadaptationofpriors.Tofurtherensuregeometryconsistency,weintroduceGlobalMotionAligningDiffusion,whichrefinescoarsepointtracksintometric-scale,globallyaligned3Dtrajectories.HarmoHOIenableson-the-flyco-evolutionof2Dappearanceand3Dmotionduringdenoising.Toovercomethescarcityofmulti-viewHOIdata,weemployahybriddatacurriculumlearningstrategythatsuccessfullytransfersgenericpriorsfromsingle-viewdatatosynchronizedmulti-viewgeneration.ExperimentalresultsshowthatHarmoHOIachievesstate-of-the-artperformanceinvisualquality,motionplausibility,andmulti-viewgeometricconsistency.Projectpageavailableathttps://droliven.github.io/HarmoHOI_project.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2607\.17097

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.17097 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.17097 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.17097 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

OneHOI: Unifying Human-Object Interaction Generation and Editing

Hugging Face Daily Papers

OneHOI is a unified diffusion transformer framework that consolidates human-object interaction (HOI) generation and editing into a single conditional denoising process using relational modeling and structured attention mechanisms. The approach achieves state-of-the-art results across both HOI generation and editing tasks with support for multiple control modalities.