Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs

H2RB adapts a pretrained VLA from unpaired human and robot demonstrations by aligning hand geometry in VLM conditioning and sharing action latents with embodiment-specific decoders.

Adapting pretrained VLA policies to a target robot and environment still requires costly embodiment-specific robot demonstrations, while human demonstrations are easier to collect but suffer a substantial human-robot embodiment gap. Prior methods often place human and robot data in a common action space, either through latent actions or reconstructed world-space hand motion. The paper argues that a common action space alone leaves the perceptual gap unresolved and does not address different action realization dynamics. H2RB targets adaptation from unpaired human and robot demonstrations while retaining the pretrained VLA's visual, semantic, and control priors.

H2RB first expresses human and robot motions in a shared wrist-hand parameterization with wrist position, 6D rotation, and hand joint angles, and converts predicted wrist pose to robot arm motion via inverse kinematics at deployment. It trains a variational action autoencoder with a shared temporal encoder and embodiment-specific decoders, using frozen posterior-mean latents as VLA prediction targets. The two latent distributions are coupled through position-wise centroid matching with variance and covariance regularization, while separate decoders preserve source-specific trajectory structure. For perception, a residual adapter and training-only readout ground VLM conditioning features to current wrist pose and five wrist-local fingertip positions using a masked Smooth-L1 geometry loss, without paired human-robot demonstrations or temporal correspondence. The VLA is optimized with flow matching on latent actions plus the weighted geometry objective, while the VLM backbone, tokenizer, and decoders remain frozen during this stage. At deployment, the policy generates a latent action from robot observation, proprioceptive state, and instruction, and the human decoder output is converted to robot commands.

• H2RB improves object transfer from 8.0 to 51.8 (+43.8) while retaining in-domain performance at 49.0. §3.2
• H2RB achieves a skill-transfer average of 62.7 and an in-domain Box PnP score of 81.0, whereas shared-action baselines achieve 0.0 on both target skills. §3.2
• Heterogeneous human-robot training improves OpenArm object transfer from 51.8 to 57.2 (+5.4) and skill transfer from 62.7 to 81.7 (+19.0). §3.3
• On RB-Y1 parcel-orientation, success improves from 64% with robot-only training to 72% with human-robot HT and 78% with geometry-grounded HT. §3.3
• Adding the action tokenizer improves object transfer from 34.7 to 43.3 and skill transfer from 0.0 to 49.7; cross-domain regularization reduces the normalized human-robot latent gap from 3.81 to 1.53 (59.8% reduction) and yields 51.8 object and 62.7 skill transfer. §4.2
• The human-transferred Open/Close policy reaches 7.36 cm/s versus 2.52 cm/s for the robot-only reference, with comparable task performance (62.7 vs. 65.3). §4.2

• No separate limitations section is provided; experiments cover OpenArm and RB-Y1 tasks, leaving broader embodiment and task generality untested in the paper.
• Each reported condition uses one independently trained policy, so the paper reports no variance or statistical significance across seeds (§B.4).
• The heterogeneous-training comparison measures the benefit of additional HT rather than isolating hand geometry-anchored alignment within a matched MT recipe (§D.3).

• H2RB uses geometry-anchored alignment to ground VLM features across embodiments without paired demonstrations or observation synthesis. §1
• A shared wrist-hand action representation uses wrist position, 6D rotation, and hand joint angles. §2.2
• The decoupled-decoder tokenizer uses a shared encoder and embodiment-specific decoders to preserve source-specific trajectory structure. §2.2
• The VLA is trained with flow matching on frozen latent action targets plus a geometry Smooth-L1 loss. §2.3
• Deployment generates a latent action and decodes it with the human decoder, then converts output to robot commands. §2.3
• Object transfer introduces target objects via human pick-and-place while retaining robot pick-and-place experience. §3.1
• Skill transfer introduces opening and closing interactions absent from source robot demonstrations. §3.1
• Instruction conditioning re-injects instruction features and improves skill transfer. §3.4