Solv3R: 3D Looped Transformer as a Multi-view Solver

Solv3R is a looped transformer with a stationary learned geometric update operator that continues multi-view reconstruction from its current state for offline, incremental, relocalization and streaming settings.

Most geometric foundation models predict entire geometry in a single forward pass or update a recurrent state as frames arrive sequentially, making expansion and refinement non-trivial. Conventional feed-forward GFMs use successive layer-specific transformations, tying intermediate representations to layer depth. DéjàView shares transformer weights but conditions each update on the refinement step, so one application cannot jointly update views that have received different iteration counts and accuracy degrades beyond its trained range.

Solv3R reformulates multi-view reconstruction as a learned stationary geometric solver. It replaces the classical update with a learned operator Tθ acting on the multi-view state and injects the initial image encoding at every solver call, using one transformer block with frame-wise and global attention shared across calls. It is fine-tuned from pretrained DéjàView, with step-conditioned gates replaced by learned constant gates and the updated state rescaled to a fixed RMS norm. The operator is trained on states that have received different numbers of solver calls and on perturbed states to satisfy stability, correction and continuation. Training samples 4–64 views, performs 16–40 solver calls without gradients then 8 differentiable calls, and applies feature noise or view resets with probability 0.15.

• Solv3R retains accuracy comparable to DéjàView at the same 16-call budget, and adaptive stopping with γ=0.02 and up to 64 calls improves nearly all reconstruction metrics with 3–4 additional solver calls on average. §5.1
• In incremental reconstruction, after solving four views for eight calls and introducing twelve additional views, Solv3R reconstructs the new views while improving original-view camera estimates, whereas DéjàView fails to maintain a consistent reconstruction; expanding from 8 to 64 views is comparable to re-solving from scratch with fewer solver calls and lower total runtime. §5.2
• With a single solved-map KV cache reused across all solver calls, Solv3R achieves the highest localization recall on 7-Scenes and Cambridge with lower cache storage than layer-wise or stage-specific alternatives. §5.3
• Without streaming-specific training or a separate optimization backend, Solv3R reports competitive streaming results across camera pose, video depth and 3D reconstruction with per-frame runtime and peak GPU memory. §5.4
• Ablations show input injection, state normalization and random-depth pre-calls are critical for long-horizon stability, while removing perturbation training largely preserves long-horizon accuracy but degrades joining and cached-context inference; the final Solv3R reaches pose accuracy 77.2 at 16 calls, 79.1 at 128 calls and 78.8 at 1024 calls. §5.5
• Global-attention heads in Solv3R keep correspondence accuracy after call 16 and their attention maps at call 64 remain close to call 16, while DéjàView heads lose correspondences and drop below 3% by call 64. §C.3

• Stated in §C.5: the solver may occasionally stop at a suboptimal solution; 143 of 144 windows meeting the stopping criterion stop below 5° rotation error, and the remaining window with little overlap between view groups is placed at a wrong position and orientation and kept there up to call 64.
• Stated in §5: training is compute-heavy, using approximately one week on 8 × H100 GPUs.
• Experiments leave open sensitivity to retrieval in relocalization: §5.3 uses an off-the-shelf image retrieval model and selects the 10 nearest map views per query, but the main paper does not analyze retrieval failures.
• Stated in §5.4: streaming predictions are reported at frame arrival for fair comparison even though incoming frames receive further refinement while remaining in the active window, so reported numbers do not reflect final refined states.

• Solv3R learns to continue solving multi-view geometry from its current state, and its update operator is independent of the iteration count. §Abstract
• DéjàView shares a transformer block but conditions each update on the iteration index τk, tying states to the iteration at which they were computed. §3.1
• A stationary solver has stability, correction and continuation properties; Solv3R trains its learned operator to satisfy these. §3.2
• Solv3R injects the initial image encoding at every solver call and uses one transformer block shared across solver calls. §3.3
• Stationarity is obtained by replacing step-conditioned gates with learned constant gates and rescaling the updated state to a fixed RMS norm. §3.3
• Training samples 4–64 views, performs 16–40 solver calls without gradients then 8 differentiable calls, and applies feature noise or view resets with probability 0.15. §5
• A single solved-map KV cache can be reused across all Solv3R solver calls, unlike layer-specific or stage-specific caches. §5.3
• Streaming uses anchor views and warm-started incoming views jointly refined with recent observations; frozen KV-cached views can be reactivated when inconsistencies appear. §5.4