CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

CaPtain is an open post-training pipeline (SFT on synthetic in-harness trajectories, then multi-turn RL with a privileged simulator critic) that turns a 4B open VLM into a code-as-policy robot agent operating over 22 low-level perception and motion primitives, reaching 73.8% on held-out LIBERO.

Existing code-as-policy (CaP) robot agents sidestep training: they either prompt closed frontier models without weight updates, which is costly and cannot be fine-tuned or run on onboard compute, or they delegate manipulation to pre-trained VLAs and high-level skills, so the agent stays bounded by the skill's training data and RL can at best learn to sequence it. Small open VLMs fail outright when asked to control a robot directly (Qwen3.5-4B solves none of the LIBERO episodes in the authors' harness). What is missing is a complete SFT-and-RL recipe that teaches a small open VLM to manipulate through low-level primitives with every manipulation decision made by the policy itself. Applying this at the low level is harder because decisions are spread over tens of requests per episode whose physical effects surface several turns later, and only a single outcome is observed at the end.

CaPtain trains and deploys the policy inside CaPtain-Harness, an execution environment exposing 22 low-level perception, geometric measurement, and guarded-motion primitives with no grasp, place, or open-drawer skills and no VLA; every call returns a structured text receipt of what physically happened, and the policy-facing API is unchanged across LIBERO, RoboTwin 2.0, RoboDojo, SimplerEnv, CaP-Bench, and a physical robot. Stage 1 applies SFT on in-harness trajectories generated with GPT-6 Astra, rendered in exactly the multi-turn format the policy sees at deployment. Stage 2 runs multi-turn RL (combining the streaming RL formulation of SAO with asymmetric actor-critic training) in which a simulator critic reads privileged physical state hidden from the policy and credits each request by how much it changes the predicted chance of success at a fixed time index, so requests that only perceive or verify receive near-zero credit. The critic is used only during training; at test time the policy relies on camera images and receipts alone, with no VLA, frontier model, or privileged state. Starting from Qwen3.5-4B, this yields CaPtain-4B.

• On held-out LIBERO, RL raises host success from the SFT model and strict success also improves, while false finishes and requests per episode fall. §4.1
• Paired RL gains over CaPtain-4B-SFT persist across perturbation suites, including LIBERO-Plus and LIBERO-MAX Dynamic. §4.1
• On LIBERO-PRO, CaPtain-4B surpasses several frontier-API systems, and multi-turn RL over low-level primitives yields a larger gain than RL around a frozen VLA. §4.2
• Without target-domain training, CaPtain-4B reaches high zero-shot success on SimplerEnv WidowX and CaP-Bench, while RoboTwin 2.0 and RoboDojo remain low. §4.3
• Replacing ground-truth object poses with noisy estimates in an offline sensitivity study degrades correlation with the privileged value-change signal. §5
• Compute accounting for the pipeline: RL to CaPtain-4B over 300 rounds plus SFT in H100-hours, with per-round and per-request timings. §M

• Single-benchmark RL training and bimanual control: the RL stage is trained exclusively on the 40 LIBERO tasks with a single 7-DoF arm; cross-benchmark performance remains below in-domain accuracy, and tasks requiring simultaneous bimanual coordination or tight kinematic reach on unfamiliar 6-DoF arms are not yet addressed (§5, referring to §4.3; RoboTwin 2.0 and RoboDojo reach only 22.6% and 0.8%).
• Privileged state during RL: training the simulator critic relies on exact simulator state; the offline sensitivity study degrades correlation to 0.49 under 2 cm pose noise, 0.36 under 5 cm, and 0.21 using the policy's own perception, indicating that online RL directly on physical hardware would require an external state estimator (§5).
• Interface and SFT coverage: the action space is bounded by the 22 capabilities of CaPtain-Harness and the behaviors demonstrated in the synthetic SFT trajectories; tasks requiring continuous force-controlled compliance, such as Wipe on CaP-Bench, cannot be expressed by guarded Cartesian position primitives (§5).
• Termination is not credited by the critic: after about round 340 host success stays flat while strict success falls from about 60% to 44% because the policy over-verifies instead of calling finish(), which the paper states suggests giving the finish decision its own target (§I).

• CaPtain-Harness exposes 22 low-level perception, geometric measurement, and guarded-motion primitives and nothing higher: no grasp, place, or open-drawer skills and no VLA; perception returns masks and points, and motion takes an explicit end-effector pose or gripper command. §1
• Every call returns a structured text receipt reporting what physically happened (measured poses, contact forces, why a motion stopped), and the policy-facing API is unchanged across LIBERO, RoboTwin 2.0, RoboDojo, SimplerEnv, CaP-Bench, and a physical robot. §1
• SFT is applied to in-harness trajectories generated with GPT-6 Astra, rendered in exactly the multi-turn format the policy sees at deployment; the data mix totals 25,961 rows and 861M tokens. §3
• Multi-turn RL maximizes expected return with a terminal return plus small non-positive per-request costs; the difficulty is assigning this sparse, delayed signal to the requests that caused it. §3.3
• A simulator critic reads privileged physical state the policy never sees and credits each request by how much it changes the predicted chance of success; because the change is measured at a fixed time index, perceive-only requests get near-zero credit and credit concentrates on grasps and placements. §1
• Training on the official LIBERO initial states caused layout memorization (per-object position standard deviation about 1.4 cm); SFT data is therefore collected on wide-layout starts with about 15 cm standard deviation plus randomized lighting, camera pose and materials during RL, which keeps mark_point and segment calls grounded in the current image. §D
• Gradient-noise analysis on 109 groups x 16 SFT rollouts gives a noise scale of 9.0k episodes for GRPO outcome advantages and 6.5k-6.7k for the simulator critic, motivating accumulation of K=4 rounds (1,024 episodes) per update. §H
• Failure taxonomy for zero-shot transfer is dominated by required bimanual coordination (167 of 437 RoboTwin first blockers), reach/inverse kinematics on RoboDojo (21 of 46), contact/force guard triggers, and running out of requests through re-verification. §J