Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Co-Design Bench evaluates coding agents on 273 real-user application revisions with hidden acceptance criteria, finding that strong models satisfy many criteria per task but fully complete only a minority of tasks.

Existing web-coding benchmarks cover reference reconstruction, synthetic or predefined edits, version-history edits, and repository issues, but coverage remains limited for compound revisions drawn from real coding sessions across localized multimodal edits and coordinated repository changes. Agentic execution also changes measured completion: on 150 WebCompass-edit tasks, qwen3.7-plus rises from 73.6% to 95.1% mean graded feature completion from single-shot to agentic execution. The paper motivates evaluating real-user revisions under iterative tool access with both criterion satisfaction and complete task success.

Co-Design Bench contains 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions, with 2,331 hidden expert-authored acceptance criteria. Each task pairs a runnable pre-edit application and normalized request with optional attachments and session history, and the solver edits the application under a shared harness. Multimodal Editing covers 100 localized edits with optional image, audio, video, or document inputs; Repository Editing covers 173 coordinated changes across components, routes, state, data flow, or content. Evaluation uses a judge-then-refute protocol: a fixed agentic evaluator grades hidden criteria using source diffs, rendered pages, and Playwright traces, then a separate LLM verifier challenges Pass and Partial verdicts; audits require renders and screenshots for visual claims and browser-action logs for interaction claims. Scores are task-normalized Strict Completion, Graded Completion, and Full-Pass Rate.

• The strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. §Abstract
• The leading endpoint reaches 0.754 strict completion on Multimodal Editing and 0.667 on Repository Editing, but fully passes only 41.0% and 17.9% of tasks. §5.2
• Strict-completion rankings correlate across tracks with Spearman rho=0.818, while endpoint mean strict scores span 28.7 percentage points on ME and 13.0 on RE. §5.2
• On Repository Editing, tasks with one or two edit types score 0.654 versus 0.591 for three or four types, a 6.3-point gap. §5.3
• Documents have the highest pooled strict score at 0.652, but only 18.2% of their model-task pairs fully pass, compared with 37.0% for text-only tasks. §5.3

• Tasks evaluate a fixed normalized request and do not simulate further user feedback during execution (§3.2).
• Task-group comparisons such as modality and edit breadth describe the benchmark rather than isolated causal effects because task content and composition differ (§5.3; Appendix B.2).
• A combined ranking can conceal strengths specific to an editing setting within this benchmark (§5.2).
• The Motion and Adaptation category contains only four tasks, so its scores summarize a particularly small group (§F.3).

• The benchmark has 273 tasks and 2,331 expert-authored acceptance criteria across Multimodal Editing and Repository Editing. §3
• Multimodal Editing contains 100 tasks with modality split 20 none, 4 image, 27 video, 32 audio, and 17 document, and it has 553 criteria with mean 5.5 per task. §3.3
• Repository Editing contains 173 tasks and 1,778 criteria, and 166 of 173 tasks require multiple edit types. §3.4
• A separate LLM verifier challenges Pass and Partial verdicts for stubs, hardcoding, unreachable entry points, varied-input failures, and request mismatches; evidence audits abort evaluation on violations. §4.1
• Strict Completion counts only Pass verdicts, Graded Completion gives Partial 0.5 credit, and Full-Pass Rate requires all criteria in a task to pass. §4.2
• Ten model endpoints are evaluated on 273 tasks under fixed Claude Code harness, container manifests, prompts, and tools, with runs capped at three hours and 120 turns. §5.1
• gpt-6-astra has the highest combined strict completion 0.699 and full-pass rate 26.4%. §5.2
• Ablations cover execution regimes, evaluator sensitivity, and scaffold robustness; single-shot versus agentic execution is compared on 150 WebCompass-edit tasks. §5.4