LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation

Hugging Face Daily Papers Papers
video-generation novel-view-synthesis egocentric-vision diffusion-models computer-vision

Summary

LEGO 提出一种无提升（lifting-free）的方法，用 LVSM 风格的 Transformer 直接合成第一人称视角，替代传统的深度估计与点云重投影管线，并将结果作为条件输入视频扩散模型，在跨视角视频生成上超越现有 SOTA 且能跨数据集泛化。

Generating an egocentric video from a single exocentric recording is a challenging case of novel view synthesis, as the two cameras share little overlap and much of the target view is unobserved. Current state-of-the-art methods reconstruct the scene explicitly by estimating depth, lifting the video into a point cloud, and re-rendering it from the egocentric camera to condition a video diffusion model. This deterministic mapping assigns each pixel to a single reprojected location, which preserves texture but translates depth errors into misplaced content. We ask what a video diffusion model should receive as its condition and propose a lifting-free answer: a learned view synthesizer, an LVSM-style transformer fine-tuned to render the egocentric view directly without depth, point clouds, or reprojection, resolving cross-view correspondence internally. In contrast, its probabilistic mapping averages each region over candidate source locations according to a learned correspondence distribution, preserving structure while fine texture is averaged away. We argue that this trade-off suits a diffusion generator, whose denoising training excels at restoring detail, so an effective condition should prioritize structural alignment over sharpness. This distribution's concentration also yields a per-region confidence, used both to mask low-confidence regions and to guide the generator toward high-confidence areas during early layout-forming denoising steps. Our approach consistently outperforms the state-of-the-art explicit pipeline and generalizes to other datasets without retraining. The synthesizer thus supplies view structure, and the diffusion model its detail.
Original Article
View Cached Full Text

Cached at: 10/09/26, 06:26 AM

Paper page - LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation

Source: https://huggingface.co/papers/2610.12442

Abstract

Generatinganegocentricvideofromasingleexocentricrecordingisachallengingcaseofnovelviewsynthesis,asthetwocamerassharelittleoverlapandmuchofthetargetviewisunobserved.Currentstate-of-the-artmethodsreconstructthesceneexplicitlybyestimatingdepth,liftingthevideointoapointcloud,andre-renderingitfromtheegocentriccameratoconditionavideodiffusionmodel.Thisdeterministicmappingassignseachpixeltoasinglereprojectedlocation,whichpreservestexturebuttranslatesdeptherrorsintomisplacedcontent.Weaskwhatavideodiffusionmodelshouldreceiveasitsconditionandproposealifting-freeanswer:alearnedviewsynthesizer,anLVSM-styletransformerfine-tunedtorendertheegocentricviewdirectlywithoutdepth,pointclouds,orreprojection,resolvingcross-viewcorrespondenceinternally.Incontrast,itsprobabilisticmappingaverageseachregionovercandidatesourcelocationsaccordingtoalearnedcorrespondencedistribution,preservingstructurewhilefinetextureisaveragedaway.Wearguethatthistrade-offsuitsadiffusiongenerator,whosedenoisingtrainingexcelsatrestoringdetail,soaneffectiveconditionshouldprioritizestructuralalignmentoversharpness.Thisdistribution’sconcentrationalsoyieldsaper-regionconfidence,usedbothtomasklow-confidenceregionsandtoguidethegeneratortowardhigh-confidenceareasduringearlylayout-formingdenoisingsteps.Ourapproachconsistentlyoutperformsthestate-of-the-artexplicitpipelineandgeneralizestootherdatasetswithoutretraining.Thesynthesizerthussuppliesviewstructure,andthediffusionmodelitsdetail.

View arXiv pageView PDFProject pageGitHub14Add to collection

Models citing this paper1

#### suhwan-cho/lego Updatedabout 1 hour ago • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.12442 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

G^3VLA: Geometric Inductive Bias for Vision-Language-Action Models

Hugging Face Daily Papers

G³VLA 通过光线嵌入、投影位置编码和跨视角融合，将标定的多相机几何结构注入预训练视觉-语言-动作模型，在不改变动作空间的情况下提升机器人操作的几何一致性与视角鲁棒性。

Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

Hugging Face Daily Papers

PRISM is a real-to-sim-to-real framework that amplifies a few real human-object interaction videos into hundreds of diverse counterfactual videos via V2V generation, then reconstructs physically plausible motions to train a generalizable humanoid loco-manipulation policy deployed on a real robot without real-world fine-tuning.

LEGO-Anything: Coding Agents for 3D Scene Reconstruction

Hugging Face Daily Papers

LEGO-Anything presents an image-to-code framework where coding agents iteratively write Blender code to reconstruct 3D scenes from single images, evaluated on the new LEGO-Bench benchmark, with LEGO-Plugin enhancing performance.