ReLift: Dense 3D Tracking as Lifted Trajectory Repair

ReLift combines frozen 2D tracks and pretrained geometry into lifted 3D trajectories, then uses a feed-forward trajectory corrector trained on simulated lifts to repair errors and complete occluded motion.

Dense 3D tracking from monocular video requires correspondence, depth, and occlusion inference, and existing methods usually learn these jointly despite scarce 4D supervision. Pretrained 2D trackers and geometry models already provide strong correspondences and depth and cameras. Lifted trajectories with occluded segments simply interpolated can already rival dedicated 3D trackers using the same geometry. However, lifts inherit correspondence and depth errors, and under occlusion their depth comes from the occluder rather than the tracked point.

Given monocular video, frozen pretrained models supply 2D tracks, visibility, depth, and cameras; viewing rays and ranges are combined into lifted 3D trajectories in reference camera coordinates. Lifted positions are retained only when range is positive and tracker visibility exceeds a tracker-specific threshold; otherwise they are masked and gaps are initialized by interpolation. A corrector processes groups of P tracks over all T frames, alternating temporal attention along each track with across-track attention within temporal chunks, and predicts unrestricted 3D residuals added to initialized trajectories. A shared scale restores original scale, and tracker visibility scores are returned unchanged. The corrector is trained on simulated lifts made by randomized geometric corruptions of clean PointOdyssey trajectories, without running a tracker, depth model, or renderer. Inference uses C=4 frame chunks and P=1024 tracks per group.

• On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points. §Abstract
• Using the same depth and camera estimates as 4RC, CoWTracker + 4RC obtains mean 89.01 on all points and 86.95 on dynamic points, versus 4RC at 86.94 and 81.60. §4.2
• Under global-scale alignment, correction improves all six lifts by 1.4 to 2.6 points on all points and 2.6 to 3.8 on dynamic points. §C.3
• Ray masking improves ADT APD3D by about 2.8 points overall and 8.1 on occluded points. §D.2
• On Kubric under RANSAC Sim(3), CoWTracker + 4RC reaches 84.88 on all points, 80.66 on dynamic points, and 0.324 EPE, while 4RC reaches 85.33, 78.81, and 0.299. §C.1
• At 512 squared, the AllTracker + VGGT-Omega pipeline takes 7.3 s and AllTracker + DA3 takes 11.4 s, versus 4RC at 31.8 s. §D.4

• Performance remains sensitive to input geometry, and the paper motivates future work on robustness to geometric and persistent correspondence errors.
• On Kubric under RANSAC Sim(3), the DA3 and VGGT-Omega configurations trail 4RC, though they outperform methods using the same geometry.
• Visibility accuracy comes from the 2D tracker because the corrector returns visibility scores unchanged.
• Experiments are reported on WorldTrack and Kubric only, so generalization to other datasets or domains is not established in the paper.

• Pretrained 2D tracks lifted with estimated geometry and simple gap initialization can match or exceed dedicated 3D trackers using the same geometry. §1
• Lifts have four failure patterns: correspondence errors perturb rays, depth jitter and drift displace along rays, wrong-surface depth, and missing observations. §3.1
• Observations are retained only when range is positive and visibility exceeds a tracker-specific gamma; gamma is 0.85 for AllTracker and 0.65 for CoWTracker. §A.1
• Alternating temporal and across-track attention costs O(PN^2+NP^2) per block, compared with O(P^2N^2) for joint attention over all tokens. §3.3
• Training uses PointOdyssey clean 3D trajectories with simulated lifting corruptions, without images or provider inference. §3.4
• One trained corrector improves all six evaluated tracker-geometry lifts without retraining. §1
• The corrector returns tracker visibility scores unchanged and refines positions used by AJ and EPE. §C.2
• Performance remains sensitive to input geometry, motivating robustness to geometric and persistent correspondence errors. §5