When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?

The paper finds that frozen video-world-model representations expose modest physical-label information and action sensitivity, but native prediction errors are strongly affected by appearance and static content and can rank impossible events below possible ones.

Prediction errors from video world models are often treated as physical surprise, but the same errors can reflect appearance, static content, motion, or ordinary prediction difficulty. Recoverable physical information may be present even when native error responds mainly to other video properties. The paper asks which evidence supports interpreting a model’s native prediction error as a measure of physical surprise.

The framework keeps each pipeline’s encoders, token selection, normalization, and native distance, and evaluates six checks: representation readout, layerwise geometry, action sensitivity, appearance/static controls, physical-validity ordering, and threshold transfer. It trains linear probes on frozen features and compares V-JEPA2-G, VideoMAE2-G, and PixelSummary on IntPhys2 with five-fold scene cross-fitting and scene-bootstrap intervals. It measures appearance and physical shifts across layers using relative magnitude, directional coherence, and subspace overlap. It tests action-conditioned models LeWM, DINO-WM, and JEPA-WM on TwoRoom and PushT under correct, zero, and reversed actions, and tests appearance changes in Physion++ Mass with persistent and target-only edits. Controlled Kubric crossed-support clips vary support, trajectory, and hue; a threshold fitted on separate calibration configurations is applied unchanged to evaluation clips. All comparisons retain native score magnitudes within models and use bootstrap over configurations, scenes, matched pairs, or episodes with Bonferroni-adjusted intervals for specified families.

• On IntPhys2 held-out scenes, V-JEPA2-G linear probes recover appearance labels easily and physics labels modestly above chance, with a positive paired advantage over VideoMAE2-G on the all-appearance physics task. §4.3
• In crossed-support Kubric clips, impossible-minus-possible native error is negative on solid support and positive over a gap, and the unchanged threshold falsely flags many lawful clips, including all clean lawful solid-support clips. §E.2
• In Physion++ Mass, still-frame controls have higher mean error than original clips, while persistent hue and persistent pixel darkening lower mean error relative to the original. §4.2
• Layerwise geometry shows that V-JEPA2-G appearance shifts become smaller relative to between-video distances at deeper layers, and final appearance-physics subspace overlap is low but above a dimension-matched random reference. §4.4
• All three action-conditioned models assign higher prediction error to incorrect actions for the same observed outcome, and color changes also alter error without changing simulator state. §4.6
• Adding final-encoder geometry changes physical-margin R2 only slightly, with intervals including gains and losses; no improvement interval across the tested locations and tasks lies entirely above zero. §4.5

• The controlled support experiment samples one procedural family with restricted cameras, colors, and lighting, so real-world and material generalization are untested.
• The IntPhys2 readout and geometry-to-error transfer are post-hoc analyses whose intervals condition on fitted models and omit refitting uncertainty, and the readout also conditions on fold assignments.
• Model comparisons vary architecture, pretraining, capacity, and native score, so they do not isolate the JEPA objective or establish a general limit on planning or anomaly-detection performance.
• Physion++ provides contact labels and no physical-violation labels, and its full-clip scoring implementation permits target information to affect context tokens.

• The evaluation framework separates frozen-representation readout, layerwise geometry, action sensitivity, appearance/static dependence, physical-validity ordering, and threshold transfer. §3
• On IntPhys2 held-out scenes, V-JEPA2-G linear probes reach 57.9% physics accuracy and 97.7% appearance accuracy. §4.3
• In crossed-support Kubric clips, stopping has higher mean error than falling in both support settings, so impossible-minus-possible ordering reverses between solid support and a gap. §4.1
• A threshold fitted on separate clean calibration configurations gives 53.9% balanced accuracy and falsely flags about 69% of lawful clips, including every lawful solid-support continuation. §4.1
• In Physion++ Mass, still-frame controls have higher mean error than originals, and persistent hue or pixel darkening can lower mean error relative to original. §4.2
• V-JEPA2-G’s hue relative magnitude falls from about 57% to 8%, and final appearance-physics subspace overlap is about 6% versus a 0.28% random reference. §4.4
• Incorrect actions raise prediction error in all three action-conditioned models while the observed outcome is fixed, and color changes also alter error. §4.6
• Adding final-encoder geometry changes physical-margin R2 from about 0.771 to 0.774, with intervals including gains and losses and no improvement interval above zero. §4.5