Democratizing Robot-Use Agents: A Unified Harness with Open Models

uni-robot-agent is a unified, modular harness that lets open-weight models perform competitive robot use in simulation and the real world, and the paper links effective robot use to four general capabilities.

Robot-use agents have shown promising zero-shot manipulation, but reliance on frontier proprietary models limits research into further training, makes large-scale interaction costly, and restricts model-level optimization. Open-weight models face a capability gap for robot use, and prior work has often used privileged information or task-specific skills. The paper argues that a generalizable, effective harness is needed first to understand and maximize existing open-weight capabilities.

uni-robot-agent connects different agent runtimes to a shared toolkit for simulation and physical robots through environment adapters and common tool arguments, units, and feedback formats. The harness overlaps model reasoning with tool execution using background tool calls and resource-aware scheduling while preserving required operation order. It avoids privileged inputs such as simulator depth, CAD assets, hidden annotations, task-verifier feedback, and web access, using only RGB images, camera calibration, robot proprioception and descriptions, and task instructions. The system is organized around unified, efficient, realistic, and modular principles, with replaceable runtimes, tools, perception models, and robot backends. Interaction records are intended to be combinable for subsequent training.

• Qwen3.8-Flash-Next with uni-robot-agent surpasses GPT-6 Astra with RoboProbe and listed VLA/WAM baselines on RoboDojo, while DeepSeek-V4.1-Flash also outperforms the VLA/WAM baselines. §3.1.2
• Switching from uni-robot-agent to the strongest baseline harness substantially degrades Qwen3.8-Flash-Next performance. §3.1.2
• On the real-world benchmark, Qwen3.8-Flash-Next with uni-robot-agent matches fine-tuned pi_0.5 and outperforms zero-shot Dream-1; GPT-6 Astra with the harness performs best. §3.2.2
• Without depth but with camera parameters, Qwen and Astra retain much of their depth-enabled scores; without camera parameters, Qwen drops sharply and Astra decreases moderately. §5.1
• Disabling the robot skill library leaves comparable performance, indicating that robot-use performance does not depend on task-specific knowledge from skills. §5.2
• Qwen offers a favorable performance-cost trade-off, while Astra achieves the highest score at nearly 20x the cost per episode. §6.1

• The additional model comparisons across generations, families, and sizes use one seed and one layout (§4.1).
• The GPT-6 Astra depth ablation uses one seed and one layout per task due to limited evaluation costs (§5.1).
• Precise manipulation and reliable long-horizon execution remain challenges in the evaluations (§8).
• In real-world evaluation, uni-robot-agent additionally receives D405 depth, whereas pi_0.5 and Dream-1 use RGB only (§3.2.1).

• Shared tools and environment adapters avoid N×M tool-environment implementations and common request and feedback formats make cross-environment experience easier to combine for training. §2.2
• Efficient execution combines background tool calls with resource-aware scheduling to pipeline agent reasoning and tool execution while preserving required order. §2.3
• The harness aims for realistic interaction without simulator-provided depth or task-verifier feedback, relying on information available in physical deployments. §2.4
• Simulation evaluation covers 42 RoboDojo tasks with 252 episodes, using only RGB images, camera calibration, robot proprioception and descriptions, and task instructions. §3.1.1
• Trajectory analysis identifies four failure modes: general visual understanding, fine-grained spatial grounding, localization and pose computation code, and embodied reasoning. §4.3
• RoboDojo performance shows positive correlations with MMMU-Pro, ERQA, ScreenSpot-Pro, and LiveCodeBench v6, with particularly strong correlations for embodied reasoning and general visual understanding and weaker correlation for spatial grounding. §4.4
• The real-world benchmark contains 19 tasks across five equally weighted dimensions: spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks. §3.2.1
• Pipelined execution overlaps model requests with physical actions, and episode duration is compared against serial execution on five selected tasks. §6.2