RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

RealCLI is a paired CLI benchmark that evaluates autonomous execution and scripted human-agent collaboration on matched tasks, revealing low joint task-collaboration performance among frontier agents.

Existing CLI benchmarks generally give agents complete, fixed specifications, let them execute autonomously, and evaluate only final task outcomes. Long-horizon CLI work still requires users to add information, revise constraints, correct execution, and authorize high-impact actions. Prior interactive benchmarks often treat interaction as part of task completion or focus on specific collaborative behaviors, lacking a unified evaluation of both task effectiveness and multi-dimensional collaboration quality.

RealCLI pairs EXEC and COLLAB modes on 101 CLI tasks that share the task goal, execution environment, and verifier. EXEC provides the complete specification upfront, while COLLAB uses a task-level user script with information slices and interaction probes to release information progressively and trigger clarification, correction, constraint-revision, authorization, and safety scenarios. Collaboration is measured with a five-dimensional framework covering Understandability, Steerability, Observation and Recovery, Trust Calibration, and Safety and Risk Awareness, operationalized into 22 subdimensions and scenario templates plus trajectory checks. Scores aggregate from all-or-nothing task-level scenario checks to a 75% cross-task scenario threshold, then equal-weight subdimension and dimension averages to produce S_COLLAB; J_COLLAB is R_COLLAB multiplied by S_COLLAB. Tasks are built from real CLI logs through base instance synthesis, interactive script authoring, validation, piloting, and final curation. Experiments evaluate 11 models on three scaffolds for 33 configurations using avg@3, a 600-turn cap, and isolated containers.

• The best configuration attains an average score of 49.2%, and the peak joint collaboration score reaches 37.6%. §4.2
• On the Codex scaffold, GPT-5.6 Sol reaches an autonomous execution reward of 58.5% but its joint collaboration score is 8.5%, a decrease of 50.0%. §4.2
• Across 11 Claude Code configurations and 101 matched tasks, interactive and autonomous execution achieve comparable mean pass rates, with substantial bidirectional shifts in matched outcomes. §4.3
• Strict scenario pass rates are higher for updating a plan after an obstacle than for recognizing scope expansion, and higher for following successive requests than for retaining earlier constraints or handling conflicting instructions. §4.4
• Functional or boundary-case errors are the most common failure type in both modes, and most successful interactive trajectories still fail at least one applicable collaboration check. §4.5
• Most constraint-retention failures are artifact mismatches, and many artifact mismatches with applicable communication checks still pass every communication check. §4.5

• No explicit limitations section is provided; the benchmark uses a scripted user simulator, GLM-5.3, rather than human users, so real-user behavior is not evaluated.
• Collaboration scoring relies on an LLM judge using Claude Code with Claude-Opus-4.8 plus simulator audits; 98.0% of audited trajectories receive a no-harmful-deviation judgment, leaving judge and simulator deviations as a residual risk.
• The experiments cover 101 tasks, 11 models, and three agent scaffolds; the paper does not report evaluation on other scaffolds or with human users.

• RealCLI is a paired benchmark where EXEC supplies complete specifications upfront and COLLAB introduces a scripted user that provides progressive information, corrections, constraint revisions, and authorization decisions. §2
• The collaboration framework has five dimensions: Understandability, Steerability, Observation and Recovery, Trust Calibration, and Safety and Risk Awareness. §2.2
• Collaboration scoring uses a four-level bottom-up aggregation: all-or-nothing task-level scenario perfection, a 75% cross-task scenario consistency threshold, equal-weight subdimension and dimension averages, and a composite S_COLLAB. §2.4
• The joint collaboration score is the product of collaborative task success and composite collaboration score, so high task success cannot compensate for weak collaboration. §2.4
• Tasks are constructed by synthesizing base instances from real CLI logs and then adding an interaction layer through scenario assignment, script authoring, validation, end-to-end piloting, and final curation. §3
• Evaluation covers 33 configurations composed of 11 models and three agent scaffolds, all 101 tasks, a 600-turn cap, and avg@3 scoring. §4.1
• User interaction does not provide a uniform task-outcome boost; effects vary by scaffold and task domain, with gains concentrated in local verifiable requirement additions and regressions in tasks requiring strict consistency across constraints. §4.3
• The simulator audit reports that 98.0% of audited trajectories receive a no-harmful-deviation judgment, with deviations centering on empty or protocol messages, incomplete probe coverage, and script-intent departures. §E.1