WebHarbor: Docking Real Websites for Evolving GUI Agent Environments

WebHarbor is a suite of 153 dockerized, executable reconstructions of real websites, built through task-driven human-agent collaboration, providing 3,518 tasks for evaluating and RL-training GUI web agents under deterministic control.

GUI web agents need realistic and interactive environments that stay stable during both evaluation and training. Evaluating on live websites suffers from changing content, access restrictions, and limited coverage of authenticated workflows. Existing synthetic training environments often lack functional depth and visual fidelity. WebHarbor targets both reliable evaluation and cost-efficient reinforcement learning in high-fidelity synthetic web environments.

WebHarbor is built as a suite of 153 executable and dockerized website environments that reconstruct the visual interfaces, content, and functional workflows of real-world web services under deterministic control. Construction is task-driven human-agent collaboration: environments evolve as new tasks expand their functional scope, coding agents implement the required changes, and human contributors audit visual fidelity, functional correctness, and task validity. The resulting suite contains 3,518 tasks spanning multi-step and post-authentication workflows. Beyond evaluation, the environments are used for cost-efficient reinforcement learning. The authors train a Qwen 4B-level GUI agent with RL in WebHarbor and test generalization to unseen tasks across four benchmarks. Code and data are released publicly.

• WebHarbor comprises 153 executable, dockerized website environments reconstructing real web services under deterministic control. §Abstract
• The suite contains 3,518 tasks, including multi-step and post-authentication workflows that existing works often fail to cover. §Abstract
• A Qwen 4B-level GUI agent trained with RL in WebHarbor generalizes to unseen tasks. §Abstract
• The RL-trained agent performs competitively with leading open-source GUI web agents across four benchmarks. §Abstract

• The provided text (abstract only) states no limitations section; per-benchmark scores and margins over baseline agents are not reported, so the size of the competitive gap on the four benchmarks is left open.
• The paper describes environments as faithful reconstructions of real services, but the provided text gives no quantitative measure of visual fidelity or functional coverage per site; this is left open.
• RL is demonstrated with a single Qwen 4B-level agent, so behavior at other model scales is not covered in the provided text.
• The cost or effort of the human auditing and agent-implemented environment evolution is not quantified in the provided text.

• Live website evaluation is unstable due to changing content, access restrictions, and poor coverage of authenticated workflows. §Abstract
• Existing synthetic training environments often lack functional depth and visual fidelity. §Abstract
• Environments are executable and dockerized, giving deterministic control for evaluation and training. §Abstract
• Environments evolve as new tasks expand their functional scope, with coding agents implementing changes. §Abstract
• Human contributors audit visual fidelity, functional correctness, and task validity. §Abstract
• The 3,518 tasks span multi-step and post-authentication workflows. §Abstract
• WebHarbor supports cost-efficient reinforcement learning, not only evaluation. §Abstract
• A Qwen 4B-level agent RL-trained in WebHarbor generalizes to unseen tasks and is competitive with leading open-source GUI web agents on four benchmarks. §Abstract