ReLift: Dense 3D Tracking as Lifted Trajectory Repair

Hugging Face Daily Papers Papers
3d-tracking computer-vision monocular-video trajectory-repair geometric-learning occlusion-handling

Summary

ReLift reframes dense 3D tracking from monocular video as geometric trajectory repair, using a feed-forward trajectory corrector that alternates attention along and across lifted trajectories. It achieves state-of-the-art mean accuracy on WorldTrack, improving over the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points without image-based training.

Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. We show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core, a trajectory corrector alternates attention along and across trajectories to preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.
Original Article
View Cached Full Text

Cached at: 11/02/26, 07:27 AM

Paper page - ReLift: Dense 3D Tracking as Lifted Trajectory Repair

Source: https://huggingface.co/papers/2610.42136

Abstract

ReLift improves dense 3D tracking by repairing and completing lifted trajectories using geometric trajectory refinement without requiring image-based training.

Dense 3D trackingfrommonocular videorequires establishingpoint correspondencesover time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of4D supervision. We show that strongpretrained 2D trackersand3D geometry modelsprovide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficientfeed-forward frameworkthat recastsdense 3D trackingasgeometric trajectory repairand completion. At its core, atrajectory correctoralternatesattention along and across trajectoriesto preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating4D supervisionon trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.42136

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.42136 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.42136 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.42136 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

Hugging Face Daily Papers

TrackEverything introduces a 3D point tracker that overcomes the trade-off between sparse long-horizon and dense short-horizon tracking by representing videos as persistent 3D scene tracks with voxelization-based de-duplication, achieving state-of-the-art performance on TAPVid-3D and handling videos over 1000 frames within 40GB GPU memory.

ReferTrack: Referring Then Tracking for Embodied Visual Tracking

Hugging Face Daily Papers

ReferTrack introduces a referring-then-tracking paradigm for embodied visual tracking, achieving state-of-the-art performance on EVT-Bench with single-view success rates up to 89.4%, and demonstrating robust sim-to-real transfer on legged and humanoid robots.

Anticipatory Reinforcement Learning for Trajectory Tracking

arXiv cs.LG

This paper introduces a predictive formulation for deep reinforcement learning that augments the state space with future reference horizons to enable anticipatory control for trajectory tracking. Simulation results show significant error reduction, though zero-shot transfer to physical hardware reveals a sim-to-real gap.