PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data

DataFarm uses a coding agent, a qualified simulator, and a real-image-conditioned video model to turn one calibration episode into verified action-labelled multi-view robot data that trains real-robot policies at lower cost than teleoperation.

Teleoperation provides faithful robot data but costs robot and operator time, requires resets, and starts over for every new task. Existing synthetic pipelines trade collection cost for human engineering per task and a simulation-to-reality visual gap; real-to-sim-to-real leaves hard-to-reconstruct appearance synthetic, while video generation without exact action labels cannot supply labels. The paper asks whether a coding agent can take over the teleoperator's job and produce real-ready data cheaply for each new task.

DataFarm takes a request with one calibration episode, a natural-language task and success description, episode count, and data mode; the agent grounds the scene, cameras, and scale, labels physical quantities as measured, fitted, assumed, or unknown, and qualifies a simulator variant with skill probes. It writes a task contract and freezes a separate judge, then compiles an expert that cannot read the judge or held-out seeds; the expert is admitted only if the judge passes on holdout seeds. The simulator rolls out the expert from sampled initial states to produce exact actions, states, and contacts, and a real-image-conditioned video model renders each rollout with all camera views in one generation, with appearance changes specified in text. An independent verifier calibrated on corrupted episodes and a separate reviewer compare renders with the simulated state and real references; only admitted episodes enter the dataset, and failed attempts remain in a ledger.

• Policies trained only on farm data reached 93 successes in 100 trials in the original scene versus 84 for teleoperation-trained policies; under unseen tablecloth and coloured light they kept 87 versus 46. §6.1
• The conventional real-to-sim-to-real Sim + DR baseline succeeded in 26 of 100 trials in the original scene and 21 in the unseen scene, and never completed weigh apple. §6.1
• Adapting a task averaged $21 of API usage, a render contract added $5, each further episode cost $0.80 on average, with $1.06 per episode over 100 episodes; one hour of three-view data cost $107, compared with a $117 China quote and $290 North America labour estimate. §6.2
• H3 renders deviated from simulated optical flow by 4.4 pixels and agreed on 65% of moving pixels; Sim + DR differed by 3.4 pixels with 73% agreement, and pairing the same H3 renders with simulator video half a second later raised error to 8.8 pixels and lowered agreement to 41%. §6.3
• Head-view KID to real camera frames was 0.83 for H3 and 0.88 for Seedance, versus 1.93 for the simulator and 2.50 with domain randomization. §6.4
• Calling the video model directly on one canvas had motion correlation 0.07 versus 0.74 for the farm renderer, with lag 12.7 frames and 11.2 jumps per 1,000 frames; the farm renderer had 2.9 jumps. §6.4

• The farm depends on a simulator that can reproduce each task's skills; simulator qualification detects failure, but the task then needs a new simulator variant before data can be produced (§7).
• A calibration episode cannot identify dynamics it does not exhibit, such as mass or friction; the farm labels these as assumed and randomizes over them, which may not suffice for tasks whose success depends on precise dynamics (§7).
• Generated-video verification is imperfect: rendered episodes can contain defects that automatic checks miss, which is why every episode is independently reviewed and rejected renders count toward cost (§7).
• Evaluation covers one dual-arm embodiment and five tasks; other embodiments and task families remain to be tested (§7).

• A request contains a calibration episode, a natural-language task and success description, episode count, and data mode sim2real or ood(V); no task adapter, judge, or prompt exists in advance. §3
• The capability layer is shared and version-pinned; the agent generates task artifacts including grounding, contract, sampler, expert, judge, render contract, and batch configurations. §3.1
• Average farm cost per admitted episode is (C_adapt+C_contract)/N+m and falls toward marginal cost m as one-time costs are spread; failed attempts enter through yield. §3.2
• Grounding labels each physical quantity as measured, fitted, assumed, or unknown and randomizes uncertain quantities rather than guessing them. §4.2
• Simulator qualification runs skill probes over candidate variants and selects the variant with the fewest changes that passes every probe, bound by a certificate. §4.3
• The judge is frozen before the expert is written; the expert author can neither read the judge nor run held-out admission seeds, and admitted trajectories must be single continuous executions. §4.4
• The render contract states target appearance, permitted variations, and real references per camera; authorized variation never extends to the robot, object count, or camera layout. §4.5
• Verifier specifications are calibrated by mutation testing on deliberately corrupted episodes; every episode also gets independent review comparing decoded video with references and simulated state. §4.6