DreamX: Video Diffusion Model is a Universal Latent Action Learner

DreamX repurposes a pretrained video diffusion model to extract compact latent actions that reconstruct future video and transfer across human and robot embodiments.

Collecting diverse robot demonstrations at scale is costly, and action supervision remains tied to the embodiment that produces it. Human videos lack robot action labels, and even robot demonstrations use different control spaces across embodiments. Latent action models infer actions from visual transitions, but reconstruction accuracy alone does not ensure latent actions isolate the underlying action from viewpoint and scene appearance. The challenge is capturing action-relevant structure shared across embodiments while separating it from domain-specific visual factors.

DreamX initializes from Cosmos-Predict2.5 2B, freezes the backbone, and trains rank-16 LoRA modules plus query embeddings. Learnable queries pass alongside video tokens through spatiotemporal attention to summarize observed dynamics into latent actions. Each latent video frame is compressed by 300× from 1,200 frame tokens to 4 latent-action tokens. These latent actions replace text conditioning, and a decoder initialized from the same backbone reconstructs future frames from the latent actions and initial frame, with auxiliary action supervision when labels are available. Motion transfer extracts source latent actions with the frozen encoder and decodes them with a target initial frame or target robot state. For policy learning, a variational bottleneck compresses latent actions into codes Z, and a policy predicts Z plus low-level actions through a DiT action head without generating future video during execution.

• On LARYBench, DreamX achieves the best performance across all benchmarks; compared with DiLA it reduces average regression MSE and increases average classification accuracy. §4.1
• The 32-channel compressed representation FAE-32 also outperforms all baselines on LARYBench, with average regression MSE and classification accuracy of 0.22 and 67.15%. §4.1
• In real-world drawer closing, DreamX reports higher average subgoal success and much lower inference latency than Mimic-Video. §4.4
• Auxiliary action supervision slightly improves regression MSE but slightly decreases classification accuracy, suggesting DreamX captures motion and action semantics from video alone. §4.5

• Paper states that cross-embodiment transfer and robot policies may fail outside the evaluated settings and require safety validation before deployment.
• Experiments leave open broader policy generalization: the real-world policy study is a drawer-closing task, and the paper describes it as demonstrating feasibility in the evaluated setting.
• Experiments leave open closed-loop action transfer: action transfer is executed open-loop without visual observations, with proprioception providing only the robot's initial state.

• DreamX uses a pretrained video diffusion backbone to extract latent actions with learnable queries that gather action-relevant information across frames. §3.2
• Queries condense 1,200 frame tokens into 4 latent-action tokens, a 300× reduction. §3.2
• Latent actions replace text conditioning, and future frames are reconstructed from the latent actions and initial frame with auxiliary action supervision when labels exist. §3.2
• Motion transfer reuses source latent actions with a target initial frame or target robot state without paired source-target data. §3.3
• The DreamX policy predicts compact latent-action codes Z and low-level actions through a DiT action head, avoiding future-video generation at execution. §3.4
• Pretraining uses 2.19k hours of human and robot video from six datasets; human demonstrations contribute 469 hours. §A.1
• DreamX trains for 78,000 steps at 480×640 resolution with separate rank-16 LoRA modules and up to 32 query tokens per clip. §A.2
• Zero-shot human-to-robot video transfer is evaluated on 1,008 clips across 21 tasks, with 19 generated frames per clip. §B.5