HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience

HuGe trains humanoid generalist VLAs on over 2,000 hours of egocentric human and robot data through a three-stage pipeline, using SONIC motion-token actions and data adaptation to achieve state-of-the-art HumanoidArena results and improved real-world G1 pick-and-drop success.

Extending VLAs to humanoid robots requires coordinated whole-body control and diverse demonstrations that are costly to collect on real robots. Egocentric human demonstrations offer a complementary source of task experience, but visual embodiment differences and missing robot proprioception complicate transfer to humanoid policies. The paper addresses how to train generalist humanoid VLAs that translate language instructions into coordinated whole-body behavior using egocentric human and robot data.

HuGe predicts a 66-dimensional action vector composed of a 64-dimensional latent motion token for the pretrained SONIC whole-body controller and two binary hand-control signals. It instantiates three existing VLA designs—OFT, GR00T, and Pi—with Qwen-based vision-language backbones, without adding modules or modifying architectural designs. The training recipe has three stages: egocentric pre-training, humanoid adaptation on real G1 demonstrations, and downstream task fine-tuning. During pre-training, simulation playback constructs 34-dimensional pseudo-proprioceptive states for egocentric data; during downstream fine-tuning, the Phantom visual editing pipeline removes human hands and arms, inpaints regions, and composites rendered robotic hands. The method is evaluated on seven HumanoidArena tasks and a real-world Unitree G1 cola pick-and-drop task.

• Full-parameter fine-tuning achieves average success rates of 55.00%, 58.33%, and 60.24% for HuGe-OFT, HuGe-GR00T, and HuGe-Pi, exceeding LoRA by 19.29–38.81 percentage points. §4.1.1
• Adding pseudo-proprioceptive states in egocentric pre-training improves average success by 5.00 and 2.86 percentage points for HuGe-GR00T and HuGe-Pi; SitSofa rises from 61.67% to 78.33% and 68.33% to 83.33%, while HuGe-OFT decreases from 56.43% to 54.05%. §4.1.2
• With the full egocentric pre-training corpus, HuGe-GR00T and HuGe-Pi reach average performance of 66.67% and 65.00%, while HuGe-OFT declines at the larger scale. §4.1.3
• HuGe-GR00T and HuGe-Pi achieve overall averages of 66.67% and 65.00%, exceeding strongest baseline π0.5 by 12.86 and 11.19 percentage points; HuGe-OFT reaches 54.05% and leads Boxing and VisNavi with 73.33% each; all variants score between 0% and 15.00% on DoubleDesk. §4.1.4
• Visual adaptation reduces Fréchet distance from raw to visually adapted egocentric features relative to real robot features: 2.0311 to 1.3499 (33.54%) for Wan2.2, 7.5697 to 5.5204 (27.07%) for VideoMAE V2, and 305.5143 to 263.2697 (13.83%) for V-JEPA 2. §4.2.1
• On real G1 cola pick-and-drop, 110 robot demonstrations alone reach 43.3% (13/30); adding 400 raw egocentric demonstrations raises success to 60.0% (18/30), and visually adapted egocentric demonstrations raise it to 66.7% (20/30); bottle pickup accounts for five of ten failed trials. §4.2.2

• HuGe-OFT decreases from 56.43% to 54.05% with pseudo-proprioceptive states because proprioceptive states are serialized as text and input truncation leaves incomplete descriptions; it also declines at larger egocentric pre-training scale (§4.1.2, §4.1.3).
• All three variants achieve only 0% to 15.00% success on DoubleDesk; the paper links this to repeated grasping attempts in demonstrations and whole-body data emphasis, and suggests adding large-scale robotic arm manipulation data (§4.1.4).
• Real-world evaluation uses only the π-style architecture and a single cola pick-and-drop task, so real-world generality across variants and tasks remains untested (experiments leave open; §4.2).
• During deployment, balance instability during bottle grasping is attributed in part to dexterous hand weight and contributes to unsuccessful grasp attempts (§4.2.2).

• HuGe predicts a 66-dimensional action vector made of a 64-dimensional latent motion token for SONIC and two binary hand-control signals. §3.1
• HuGe is instantiated with three existing VLA designs—OFT, GR00T, and Pi—using Qwen-based vision-language backbones; no additional modules or architectural modifications are introduced. §2.1
• The training corpus combines egocentric human and robot data across three stages; the abstract reports over 2,000 hours, while Table 1 lists Stage 1 total 1,669.610 hours and Stage 2 total 417.580 hours. §3.2
• Stage 1 uses EgoDex, EgoVerse, Xperience-10M, and BEHAVIOR-1K; it constructs a 34-dimensional pseudo-proprioceptive state from 29 joint angles, two hand-state entries, and three-dimensional gravity, and trains 50-step action chunks at 50 Hz. §3.2.1
• Stage 2 continues training on real G1 demonstrations from Humanoid Everyday, UniFoLM-WBT, and HIW-500, jointly adapting the vision-language backbone and action prediction modules. §3.2.2
• Stage 3 fine-tunes separately for HumanoidArena and the real cola task; it supplements 110 robot demonstrations with 400 egocentric demonstrations and uses Phantom visual editing to remove human hands and arms, inpaint, and composite rendered robotic hands. §3.2.3
• HumanoidArena evaluation covers seven tasks, 100 demonstrations per task, and 60 episodes per task from three seed groups of 20. §4.1
• Real-world cola pick-and-drop uses a Unitree G1 with Inspire dexterous hands and compares 110 robot demonstrations alone with augmentation by 400 raw or visually adapted egocentric demonstrations, with 30 trials per configuration. §4.2