RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild

RoboColosseum is a distributed in-the-wild robot benchmark that evaluates VLA policies and frontier agents across four embodiments via Open and Fine-tune tracks, revealing weak cross-embodiment ranking agreement and sample-efficient active matchmaking.

General-purpose robot policies must work across robots, tasks, and environments, but physical trials that reveal this variation are difficult to reproduce. Outcomes depend jointly on the policy, robot setup, initial conditions, and surrounding environment, and even nominally identical tasks can differ in objects, cameras, lighting, and physical properties. Simulation supports scale but may not predict real-world outcomes, while physical evaluation requires substantial labor for setup, resets, maintenance, and trial execution.

RoboColosseum combines unified policy serving and standardized protocols across locally operated robots in diverse environments. It initially supports four embodiments and eight AI systems: five vision-language-action policies and three frontier LLM agents. The Open Track evaluates models without task-specific adaptation in blinded, pairwise trials under matched local conditions within each comparison. The Fine-tune Track gives each policy the same 100 demonstrations per task and evaluates on 20 tasks using common success and progress criteria. The initial study ran from 17 September to 28 October 2026 and includes active matchmaking for Open Track ranking.

• Across 5,699 Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, though all systems struggle to complete open-world tasks reliably. §Abstract
• After fine-tuning with the same demonstrations, GR00T N1.7 leads on SO-101 and pi_0.5 leads on Franka and Bimanual-YAM. §Abstract
• Policy rankings show weak agreement across embodiment-specific task suites, with Kendall’s W=0.24. §Abstract
• On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. §Abstract
• Active matchmaking reaches approximately 90% agreement with the full-data ranking after 400–500 evaluations, compared with roughly 1,000 under uniform sampling. §1
• The initial study includes Fine-tune Track evaluations on 20 tasks across four embodiments. §1

• Open Track matches robot, task instruction, objects, initial setup, and timeout only within each pairwise comparison; task and environment can vary across evaluations (§3.2).
• Fine-tune Track allows training method to differ across models because policy architectures require different fine-tuning procedures, so training procedure is not fixed (§3.2).
• Agent evaluations cover SO-101, DROID, and Bimanual-YAM; experiments leave G1 out of the three frontier-agent evaluation (§10.1).
• Broader task datasets, more in-the-wild environments, and broader embodiments are pledged as future expansion rather than completed in the initial study (§6).

• RoboColosseum connects distributed evaluation sites through shared robot interfaces, policy-serving infrastructure, and two complementary tracks. §1
• It initially supports SO-101, Franka in the DROID configuration, Bimanual-YAM, and Unitree G1, with standardized observation and action specifications within each embodiment. §3.1
• The Fine-tune Track gives each policy the same 100 demonstrations per task and uses common success and progress criteria. §1
• The Open Track evaluates without task-specific adaptation through blinded, pairwise comparisons under matched local conditions. §1
• Evaluations were conducted over 41 days, from 17 September 2026 to 28 October 2026. §4
• Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. §Abstract
• Measuring partial progress reveals differences that task success alone misses; on G1 average progress ranges from 13.3% to 23.2% while no policy completes any of the five tasks. §Abstract
• Active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. §Abstract