From Pixels to Actions - Towards Native Vision-Language-Action Policies

NEO-VLA is an encoder-free vision-language-action policy that lets a flow-matching action expert read cached keys and values from multiple backbone depths, and it shows that early visual representations are valuable for control.

Most VLA models compress images through a pretrained vision encoder before action reasoning, exposing only a fixed visual endpoint. The paper argues manipulation depends on local geometry, boundaries, depth, and subtle state changes that may be vulnerable to this abstraction. It asks whether action policies should inherit a fixed visual representation or directly access evolving visual computation.

NEO-VLA maps raw pixels into backbone tokens with a lightweight two-stage convolutional patch embedder and processes visual and language tokens jointly in a 40-layer 2B backbone. It uses native rotary position embeddings over time, height, and width, with causal attention for text and bidirectional attention within visual blocks. A separate flow-matching action expert attends to cached prefix keys and values from a configurable subset of backbone layers and to its own action tokens. The layer map can select early layers, late layers, sampled layers, or all layers; the paper compares first-28, skip-28, end-28, and full-40 settings. Actions are continuous arm and body chunks trained with flow matching; inference uses 10 Euler steps with a fixed prefix KV cache.

• With only the action expert trained, the native backbone reaches higher average LIBERO success than encoder-based Qwen3-VL and Gemma-3, and full fine-tuning raises it further. §5.1
• Directly routing expert layers to intermediate Qwen3-VL vision-tower features does not reproduce the native advantage under a frozen backbone, while full fine-tuning makes the two interfaces similar. §5.1
• In layer-map ablations, skip-28 outperforms end-28 with a frozen backbone and after full fine-tuning, favoring access to early backbone layers. §5.2
• On LIBERO-plus, NEO-VLA has the highest total among listed models with both code and weights public, with strong camera and language robustness but less pronounced gains under robot and layout changes. §5.4
• Co-training with understanding data keeps understanding scores close to baseline and yields much higher RoboTwin 2.0 clean-to-randomized success than action-only fine-tuning. §5.3
• In the real-world benchmark, NEO-VLA achieves success rates of 12/20, 17/20, and 13/20 across three tasks after fine-tuning on 50 demonstration episodes per task. §5.5.2

• The paper states that differences in pretraining and model capacity preclude attributing the backbone-comparison gains solely to encoder removal.
• The paper says LIBERO cross-paper results differ in simulator versions, camera setups, training data, and evaluation protocols, so they are benchmark context rather than controlled comparisons.
• The paper reports that NEO-VLA is especially robust to changes in how the scene is perceived, but gains are less pronounced under robot and layout changes and it remains challenged when spatial configuration or task goal changes.
• The real-world evaluation uses three tasks, 50 demonstration episodes per task, and 20 trials per task, leaving broader real-world generalization open.

• The native backbone exposes evolving visual computation across depths instead of a fixed encoder output to the action expert. §1
• Pixels enter through a two-stage convolutional embedder with stride 32, producing one visual token per 32x32 image region. §3.1
• Text tokens use causal attention while visual tokens interact bidirectionally within each visual block, so visual and linguistic representations co-evolve across layers. §3.1
• The multimodal prefix includes previous and current multi-view images, camera geometry, robot state, and a structured prompt under a fixed visual-token budget. §3.2
• The action expert reads cached keys and values from a configurable layer map, making visual readout depth an explicit architectural choice. §3.3
• Training proceeds in three stages: frozen-backbone action training, full-model robot-corpus training with understanding co-training, then full-parameter in-house action finetuning. §4.3
• Swapping early-layer activations across observations causes larger translation errors than perturbing later layers, and matched residual noise has little effect. §5.2
• Preserving multimodal understanding during post-training is linked to stronger policy generalization under scene randomization. §5.3