HiFOR: Hierarchical Foresight with Action-Guided World Grounding for Vision-Language-Action Model

HiFOR couples multi-horizon future prediction (short-term state, semantic milestone, goal) with action-guided retrieval from a structured latent world to condition a flow-matching VLA policy during denoising.

Existing foresight-based VLA models typically model short-term world dynamics or long-term task objectives in isolation, so explicit dependencies between local state evolution, semantic milestones, and final goals are not established. In addition, predicted futures are usually injected directly into the action policy as extra conditioning, without distinguishing how relevant different horizons, camera views, and spatial regions are to the current action requirement. The paper argues that relevance also evolves over task execution and over the action-generation process, motivating selective retrieval rather than unconditional conditioning.

A hierarchical planning dataset is built from existing expert demonstrations: a Planning Agent decomposes the instruction into ordered subgoals, milestone completion timestamps are localized automatically with temporal-consistency checks and confidence flagging, and multi-view DINOv2 ViT-L/14 patch features pooled to a 4x4 grid serve as visual supervision. Hierarchical Foresight Prediction defines three targets per timestep (short-term, next milestone, final goal) and predicts them with three groups of learnable future queries passed through a shared prediction module under a Hierarchical Dependency Mask that progressively expands attention visibility from short-term to goal queries; short-term prediction additionally regresses the future robot state sequence. Structure-Aware Latent World Modeling projects the predicted features into structured world tokens with horizon, view, and spatial positional embeddings, forming a 3 x V x S x d_w tensor plus spatially aggregated horizon-level and view-level features. Action-Guided Factorized World Retrieval builds queries from the action expert's hidden representation at each denoising step and performs factorized attention over horizon, view, and spatial dimensions, combining the three attention weights to aggregate world values. The retrieved future features are fused with action representations and short-term state features and injected into the action expert via residual connections; training minimizes a flow-matching action loss plus the foresight prediction loss.

• HiFOR achieves the best average success rate on RoboTwin 2.0 under the Hard setting, beating the reproduced pi_0.5 baseline by 9.05%. §4.1
• On LIBERO, HiFOR reaches the best average success rate, with the largest gain over pi_0.5 on long-horizon tasks, and the highest scores on Spatial, Goal and Long. §4.1
• In real-world evaluation on six tasks, HiFOR outperforms pi_0.5 on all tasks, with the largest improvement on block stacking. §4.2
• Ablation shows successive gains from adding short-term, milestone, and goal prediction, and a further gain from adding action-guided world retrieval; LIBERO-Long also improves over the baseline. §4.3
• A 5-step short-horizon prediction, aligned with the 5-step policy update interval on LIBERO, performs best among the tested horizons of 5, 10, and 15 steps. §4.3
• Semantic feature-based future prediction outperforms direct RGB image prediction on LIBERO-Long and LIBERO-Avg. §4.3

• The paper has no explicit limitations section; its real-world evaluation is restricted to a single AgileX Piper 6-DoF arm with a parallel-jaw gripper and two RGB cameras over six tasks with 50 rollouts per task, so generalization to other embodiments is not tested (§4.2).
• The hierarchical planning dataset depends on a Planning Agent for subgoal decomposition and on automatic milestone localization, with ambiguous or low-confidence cases flagged for manual review, so supervision quality depends on that annotation pipeline (§3.3).
• The short-horizon prediction length is only studied for 5, 10, and 15 steps, and the paper notes that longer horizons extend beyond the next prediction update (§4.3).
• Real-world training uses 100 expert demonstrations per generalization task and 200 per long-horizon task; no data-efficiency or scaling analysis is reported (§4.2).

• The paper identifies two gaps in foresight VLA: isolated modeling of short-term dynamics vs. long-term objectives, and direct injection of predictions without judging relevance to the current action. §1
• Three prediction horizons are defined per timestep: short-term target min(t+Delta, t_K), next milestone target, and goal target t_K, each supervised with precomputed DINOv2 features. §3.4
• A Hierarchical Dependency Mask makes attention visibility expand progressively from short-term to goal queries, with all query groups updated jointly in a shared prediction module. §3.4
• Short-term prediction adds a state prediction head regressing the future robot state sequence, because target-timestep visual supervision alone does not constrain fine-grained robot motion. §3.4
• Structured world tokens carry horizon, view, and spatial positional embeddings and form a 3 x V x S x d_w tensor, with horizon-level and view-level features obtained by spatial averaging. §3.5
• Retrieval queries are rebuilt from the action expert's hidden states at each denoising step, and factorized attention weights over horizon, view, and spatial levels are multiplied to aggregate world values. §3.6
• The training objective is the sum of the flow-matching action loss and the hierarchical foresight prediction loss. §3.6
• Models are trained end-to-end for 30000 steps with AdamW, global batch size 512, learning rate warmed up to 5e-5 over the first 5000 steps, gradient clipping at norm 1.0, and EMA decay 0.999. §4.1