Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

S2 is a training-free offline metric that measures nearest-neighbor overlap between sensory features and end-effector trajectories and correlates with robot task success across model families.

Robot foundation models are usually evaluated by costly closed-loop rollouts, so low-cost offline proxies are needed. Existing perception-side proxies such as VQA scores or pixel-level video quality do not measure whether a representation is organized for action. The paper tests a PRH-inspired prediction that more capable embodied models should align sensory representations with the structure of physical motion.

For sampled episodes, the sensory representation is an average-pooled image-text embedding from the model; the motor representation is the next h steps of end-effector position, orientation, and gripper state. Sensory neighbors use cosine distance, and motor neighbors use DTW. S2 compares k-nearest-neighbor sets in the two spaces, with reported scores using a centered normalized CKNNA variant with k=10 unless stated otherwise. It is training-free and uses DROID, BridgeDataV2, or ALOHA reference data.

• For 12 fine-tuned VLAs, S2 has a strong positive Pearson correlation with SimplerEnv task success. §3.1
• For 8 raw VLM backbones, S2 before VLA fine-tuning correlates with the resulting VLA success rate. §3.1
• For 6 WAMs, S2 on bimanual ALOHA correlates with RoboTwin 2.0 success rates. §3.1
• For 5 real-robot policies on 40 tasks, offline S2 correlates with physical-robot success. §3.1
• S2 correlates with task success where VQA and video-quality metrics are weak or wrong-directional. §3.2
• An embodiment-mismatch negative control drops the S2-success correlation from matched bimanual ALOHA to single-arm DROID. §3.3

• The positive relationship between S2 and task success is correlational across the tested models; the paper states it has not demonstrated that sensorimotor alignment can be used directly during training to improve task success rates.
• Models with similar S2 values can still achieve different success rates, so S2 is best used as a low-cost offline proxy before costly fine-tuning or closed-loop testing.
• The evaluation covers limited model families, benchmarks, and training recipes, few models per comparison, and a single real-robot setting.
• For real-robot policies, the reported S2 correlation uses an exploratory horizon h=110; the paper notes the prespecified h=7 gives r=+0.429 with one-tailed p=0.236.
• For frozen vision encoders, the correlation weakens as the behavior-cloning demonstration budget grows and is not positive for reinforcement learning from task rewards in the reported seed sets.

• S2 measures alignment by comparing each episode's k-nearest-neighbor set in sensory space with its k-nearest-neighbor set in motor space; greater overlap means stronger alignment. §2
• The correlation stays positive under eight varied computation settings, including reference dataset, trajectory horizon, five implementation choices, and DTW warping. §3.3
• Across 10 horizons, S2 from joint vision-text and vision features correlates significantly with SimplerEnv success, while text-feature S2 is not significant under Pearson r. §C.1
• For 14 frozen vision backbones, S2 correlates with downstream success under low-data behavior cloning and visual-reward learning, but weakens as the demonstration budget grows. §B.6
• After VLA fine-tuning, vision-feature S2 correlates strongly with success while text-feature S2 does not; raw VLMs instead correlate most strongly with joint features. §4.1
• Freezing the vision encoder reduces S2 by 41% and SimplerEnv success rate by 40% in the compared Qwen2.5-VL-3B VLAs. §4.2
• For 3 WAMs, S2 is already high at the earliest denoising step and remains nearly flat, varying by about 6% at most across the schedule. §4.3
• In 3 WAMs with a 30-block video backbone, Block 14 yields the highest S2, in the mid-depth range. §4.4