Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

HCU is a unified agent interface that exposes programmatic, semantic accessibility, and visual GUI channels, letting models choose and combine them with cross-channel guards and a usage skill.

Most computer-use interfaces mirror human operation through screenshots and mouse-and-keyboard actions, imposing a human-interaction prior and underexposing semantic and programmatic capabilities. Accessibility can identify controls without pixel search, but its coverage is application-dependent, targets can change or disappear, and invocation does not establish the intended state change. Code and command-line tools can process files and data directly, while GUI interaction supplies visual evidence where structured access is insufficient.

HCU combines three channels in one agent task context: programmatic code, shell, and file tools; semantic accessibility-tree interaction; and visual GUI screenshots and input. The AX and GUI tools are exposed through MCP, with the model directing channel selection inside its existing observation-action loop. The AX channel returns compact observations, executes batched actions with optional wait steps, and reports per-target effects and AX diffs. Cross-channel action guards resolve AX references and check GUI layouts against live state immediately before input, rejecting mismatches without sending input. A supplementary skill explains channel capabilities, efficient composition, and verification or recovery, but contains no task-specific solutions.

• On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among compared systems: 72.37% Partial with GPT-6 Astra and 35.19% Binary with Claude Opus 5. §Main Results
• With GPT-6 Astra, HCU outperforms the matched CUA Driver baseline by 3.55 points in Partial and 0.90 in Binary with 29.81% fewer tool calls at nearly identical Cost. §Main Results
• With Claude Opus 5, HCU scores 3.75 points higher in Partial and 3.76 points higher in Binary than the official leaderboard result, at 20.26% lower Cost. §Main Results
• In GPT-5.4 OSWorld 1.0 ablations, HCU scores 65.59 with 28.60 Steps, versus 62.79 and 33.28 without AX, 64.00 and 33.47 without skill, and CUA Driver 62.30 and 40.74. §Interface Components
• On OSWorld 1.0, HCU improves Score by +3.29, +2.94, +12.90, and +6.81 percentage points for GPT-5.4, Claude Opus 5.5, Gemini 3.8 Flash, and Grok 4.7 versus CUA Driver. §Agent Behavior Across Models
• The skill reduces the proportion of AX and GUI calls rejected for malformed arguments from 17.37% to 1.36%. §Abstract

• AX coverage is application-dependent: the paper states that custom-drawn content may have no accessible elements, accessible objects need not map one-to-one to widgets, and virtualized items may be absent until rendered; HCU marks unavailable information and lets the model request screenshots when semantic evidence is insufficient.
• HCU prioritizes completing the requested action with native AX operations or focused input; unlike CUA Driver's best-effort background input, this design does not prioritize preserving the user's active desktop.
• The GUI action vocabulary affects performance: Claude Opus 5.5 gained 0.82 points with the Claude-style vocabulary, while GPT-5.4 scored 2.03 points higher with the default GPT-style vocabulary, so results depend on matching a model's native calling conventions.

• HCU combines programmatic tools, semantic AX interaction, and visual GUI interaction through MCP within one model-directed observation-action loop. §Hybrid Computer Use Interface
• The AX channel is designed around compact views, batched actions with optional wait steps, and post-action AX diffs so the model needs fewer round trips. §Accessibility for Computer Control
• Cross-channel action guards check AX target identity and GUI layout against live state before input and reject mismatches before sending input. §Guarding Actions Across Channels
• The supplementary skill covers channel capabilities, efficient interaction, and verification or recovery, and contains no task-specific solutions. §Skill for Effective Interface Use
• Across four models on OSWorld 1.0, HCU improves Score while changing Steps, Input, Time, and Cost relative to CUA Driver, with model-specific channel allocation. §Agent Behavior Across Models
• The paper reports that HCU outperforms CUA Driver across four backbone models in their own harnesses, averaging 6.48 more points in Score, 17% fewer Steps, and 30% lower Cost. §Introduction
• Within tasks, channels often play complementary roles, with screenshots supporting observation and AX enabling execution. §Introduction
• The default GPT-style GUI vocabulary performs better for GPT-5.4, while the Claude-style vocabulary performs better for Claude Opus 5.5. §GUI Action-Space Replacement