Probing Spatial State Representations in Visual Foundation Models

The paper functionally probes ten frozen visual foundation models across four spatial capabilities and finds that spatial readout, use of history and motion, predictive transfer, and goal-conditioned action utility dissociate across tasks, layers, and prefix lengths.

Spatial perception involves more than recognizing individual images: an observer sees only part of the world and must maintain a state beyond the current view. The paper asks whether visual foundation models provide representations that support four spatial capabilities under streaming egocentric observations: spatial structure, object persistence, predictive spatial update, and actionability. It further tests whether readout depends on visual history and motion and whether goal information has functional utility for action.

The authors use posed ScanNet++ trajectories with RGB observations, camera poses, instance geometry, and visibility annotations, with disjoint training and validation scenes. They extract frozen features from ten representations and train readouts shared within model and task across layers and prefixes. Causal prefixes encode frames 1:p independently for p in 8,12,16,24; L3 and L4 use future frames at h in 1,2,4 as targets or goals. Tasks measure metric geometry, instance identity, view consistency, object persistence, direct prediction, path integration, and goal-conditioned action in candidate-selection and continuous-output settings. Matched controls alter motion, history, transformation order, generation coordinate, or goals while keeping the relevant predictor and target fixed. Main scores average over evaluated prefixes at designated layers, with paired scene-bootstrap intervals over 69 validation scenes.

• Table 2 reports Fast3R camera-pose AUC@30 0.3870 and object-persistence angular error 32.79; V-JEPA2 path-integration global R@1 is 0.1588, and DINOv2 direct-prediction global R@1 is 0.2373. §4.1
• V-JEPA2 shows a motion gain of 18.84 pp and a history benefit of 10.26 under matched controls. §4.3
• At horizon two, normalized-depth MAE rises from 0.182 on real features to 0.246 on native predictions; retraining reaches 0.223, better than the 0.241 layout prior but worse than the real-feature readout. §4.4
• Candidate-selection direct scores exceed shuffled-goal scores for DINOv2 76.2±0.8 vs 55.5±0.8, Fast3R 79.9±1.3 vs 62.4±0.5, V-JEPA2 78.1±0.7 vs 62.0±1.1, and Wan 75.9±0.9 vs 63.7±0.5. §4.5
• In continuous output, FULL minus CURRENT-ONLY goal reduction is negative for DINOv2 -0.008 [-0.013,-0.004], V-JEPA2 -0.009 [-0.015,-0.002], and Wan -0.011 [-0.017,-0.004]; Fast3R is -0.005 [-0.012,+0.002]. §A.10

• The conclusions concern frozen feature interfaces, trained probes, and offline indoor trajectories; independently encoded causal prefixes characterize what a representation of the observed history affords rather than an incremental state update, and retrieval scores in different target feature spaces do not share a physical error scale (§6).
• Comparisons across prefix lengths do not isolate added history because a longer prefix also moves the endpoint (§2.2).
• Fast3R's training data include ScanNet++ and some validation scenes are among its training scenes, so its leading camera-pose, object-persistence, and action results may partly reflect familiarity (§3.1).
• MiniMax-H3 direct prediction uses only h=1 and is evaluated on 171 of 276 validation windows, and it is not ranked (§4.1).

• The paper defines four levels: L1 spatial structure, L2 object persistence, L3 predictive spatial update, and L4 actionability. §1
• It probes ten frozen representations spanning image encoders, video encoders, 3D models, multimodal models, video generators, and world models. §3.2
• Causal prefixes are encoded independently at p in 8,12,16,24; L3 and L4 future targets use h in 1,2,4, with MiniMax-H3 direct prediction restricted to h=1. §2.2
• Main scores average over evaluated prefixes at designated layers; no best-layer or best-prefix selection enters the aggregation. §4.1
• Matched controls keep the predictor and targets fixed while changing motion, history, transformation order, or goals. §1
• V-JEPA2 shows motion gain 18.84 pp and history benefit 10.26; VideoMAE motion gain is 10.51 pp, and Wan history benefit is 10.01. §4.3
• Native V-JEPA2 prediction retains recoverable spatial structure, but a reader trained on real features does not transfer unchanged. §4.4
• The levels are linked by two chains: camera-pose readout supports object persistence and actionability, whereas feature prediction builds on visible-scene readout and is unrelated to persistence. §6