WorldBridge: Bridging Video and Action for Task Generalization in World Action Models

WorldBridge is a World Action Model that couples separate video and action experts through structured attention and joint flow matching, and its supervision study shows OOD videos raise RoboTwin 2.0 OOD success from 25.07% to 55.90% with sparse action labels further raising it to 70.70%.

Task generalization in embodied models requires executing tasks beyond those covered by training, not only robustness to changes in object arrangements, viewpoints, lighting, or appearance. VLA models transfer semantic knowledge, but understanding an instruction does not automatically provide physical competence. World Action Models couple visual prediction with action generation, but it is unclear how broader video coverage and additional action supervision each contribute and when the video-to-action connection becomes limiting.

WorldBridge is a World Action Model with a Mixture-of-Transformers architecture containing separate video and action experts that exchange information through joint self-attention; the video expert also uses cross-attention for language features. It builds conditioning context from a video VAE and T5 text encoder, samples independent video and action flow timesteps, and trains both experts with joint flow matching to predict velocity fields for future video latents and action chunks. At inference, future video latents and action chunks are initialized from Gaussian noise and denoised from tau=1 toward tau=0 while the conditioning context remains fixed. Two-scale visual memory uses short-term and long-term observation sequences encoded with a causal video VAE and prepended to noisy future video tokens. Action forcing assigns independent flow timesteps to action groups during training so the action expert sees mixed noise levels, while inference denoises the full chunk with a shared action flow timestep. Stage-specific attention masks preserve action-independent video generation during pretraining and route memory and current observations through future-video features to action predictions during post-training, and a unified pipeline maps robot and egocentric demonstrations into a 46-dimensional action representation with dimension-wise validity masks.

• On RoboTwin 2.0, training with only in-distribution video-action supervision gave 91.85% ID success but 25.07% OOD success; adding OOD videos without OOD action labels raised OOD success to 55.90% while ID success was 91.96%. §3.1
• Keeping video data fixed, one action-labeled demonstration per OOD task raised OOD success from 55.90% to 70.70%; five demonstrations reached 84.67% and 20 reached 90.03%. §3.1
• With 100 action-labeled demonstrations per OOD task, increasing OOD videos from 100 to 550 raised OOD success from 88.83% to 93.17%, a gain of 4.34 percentage points; increasing actions from 100 to 550 with 550 videos changed 93.17% to 92.83%. §3.1
• WorldBridge achieved 96.23% average success on RoboTwin 2.0 Full and 80.9% on Clean2Random, with 96.16% Clean and 96.30% Randomized in Full and 88.3% Clean and 73.4% Randomized in Clean2Random. §3.2
• On other benchmarks, WorldBridge reported 82.8% on RoboCasa GR1 TableTop, 83.0% on RMBench, 57.6% on DOMINO Level-1 with 64.6 MS, and 84.4% average on SimplerEnv-WidowX. §3.2
• In the ablation, adding action forcing, short-term memory, and long-term memory raised success from 91.28% to 95.20% on RoboTwin, from 67.50% to 78.58% on RoboCasa, and from 14.50% to 21.07% on RoboDojo. §3.2

• Video-only OOD supervision leaves a remaining ID-OOD gap, suggesting video supervision alone does not fully resolve execution challenges.
• The conclusion states that extending the video-to-action bridge to more diverse tasks with less action supervision remains future work.
• RoboCasa training required higher-resolution replay and cropping/resizing because video generation quality was important for downstream policy performance, indicating sensitivity to video quality.

• The paper separates task generalization into video generalization, which predicts visual evolution of new tasks, and action generalization, which translates predictions into actions through a video-to-action bridge serving as an IDM. §1
• WorldBridge uses a Mixture-of-Transformers with separate video and action experts, joint self-attention, video-expert language cross-attention, and joint flow matching over future video latents and action chunks. §2.1.1
• Short-term and long-term visual memory sequences are encoded with a causal video VAE and prepended to noisy future video tokens after dropping a padding latent. §2.1.2
• Action forcing assigns independent flow timesteps to action groups during training, while inference jointly denoises all action tokens with a shared action flow timestep. §2.1.3
• Stage-specific attention masks keep video tokens action-independent during pretraining and, during post-training, route memory and current observations through future-video tokens to action tokens; memory-heavy tasks use the pretraining mask. §2.1.4
• The unified action space is 46-dimensional with arm joints 14D, end-effector poses 14D, grippers 2D, dexterous hands 12D, and waist controls 4D, plus dimension-wise validity masks. §2.3.2
• Robot and egocentric pretraining contributed on RoboTwin Clean-to-Random: no pretraining averaged 62.7%, robot pretraining 77.4%, and adding egocentric pretraining 80.9%. §3.2
• WorldBridge was deployed on AgiBot G2 and ARX AC-1, fine-tuning the same jointly pretrained checkpoint for each platform and evaluating long-horizon execution, memory-dependent interaction, spatial alignment, and bimanual coordination. §1