CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents

CrossSWE is a 171-instance benchmark for cross-repository joint development, where tasks require coordinated changes across interdependent repositories and are verified by joint-necessity testing.

Existing coding-agent benchmarks mainly increase complexity within individual repositories, so the context and patch scope remain bounded by one codebase. Recent multi-repository benchmarks use other repositories only as context, while the patch and tests stay within one repository. Real-world development often spans interdependent repositories, where API, library, or protocol changes propagate across boundaries and modifying one repository may be insufficient. CrossSWE targets this gap by requiring agents to make mutually compatible changes across multiple repositories for a single task.

CrossSWE is built through a four-stage pipeline. Candidates are collected from GitHub development histories from January 2025 through September 2026 using cross-repository cues such as API evolution, dependency migration, provider-consumer, and client-server relations. Human screening retains only candidates with concrete software changes and substantive implementation changes in at least two repositories, excluding documentation, release notes, and changelog-only changes. Each retained candidate gets a Docker environment containing all participating repositories at verified base commits, with clean Git states and without fixed checkouts, gold patches, hidden tests, or fixed-version artifacts in the agent-facing image. Joint necessity is verified by reconstructing repository-specific gold patches and exhaustively testing all 2^N mixed-version configurations: target behavior must fail in the all-base and every partially patched configuration and pass only when all required patches are applied. Problem statements are transformed into self-contained natural-language specifications that remove solution-specific information and explicitly specify required cross-repository behavior.

• The strongest evaluated model, GPT-6.1 Sol, achieves only 58.48% Pass@1 on CrossSWE, below its reported 75.2% on DeepSWE v1.1. §5.1
• Other Pass@1 scores are DeepSeek V4.1 Flash 52.63%, Claude Sonnet 5.5 50.29%, Qwen3.8-Flash-Next 45.03%, and GLM-5.3 Flash 43.86%; GPT-6.1 Sol scores 60.34% on single-language and 54.55% on multi-language tasks, while DeepSeek V4.1 Flash scores 52.73% on multi-language tasks. §5.1
• Average cost per instance is $0.3496 for GPT-6.1 Sol, $0.3411 for GLM-5.3 Flash, $0.2881 for DeepSeek V4.1 Flash, and $0.2939 for Qwen3.8-Flash-Next; average steps are 32.9 and 29.2 for the closed-source models versus 152.7, 141.7, and 182.1 for the open-source models. §5.1
• In CrossSWE, 56.1% of instances modify more than ten files and 66.7% change over 200 lines, compared with 6% and 13% in SWE-bench Pro and 30% and 32% in SWE-Bench ProMax. §3.3
• Cross-repository inconsistencies are the most prevalent failure mode at 44.2%. §Abstract
• The authors derive 330 single-side modification tasks from the 165 two-repository instances, keeping both repositories in the environment but requiring modification of only one repository. §5.2

• The paper states that coverage of large-scale multi-repository development remains limited: 165 instances involve two repositories, while only six involve three or more repositories.
• The paper states that CrossSWE provides the participating repositories to focus on dependency reasoning and coordinated modification, so repository discovery and selection from a broader ecosystem are not evaluated.
• The paper states that all models use a unified mini-SWE-agent scaffold that primarily relies on Bash-based interactions, so direct validation on other shell-based frameworks remains necessary; it also imposes limits of 300 interaction steps and $10 per instance.

• CrossSWE tasks require coordinated changes across multiple repositories, and an instance is resolved only when the joint repository state satisfies the complete evaluation suite. §3.1
• The benchmark contains 171 real-world instances spanning 165 repositories, 20 technical domains, and 13 programming languages. §Abstract
• Instance composition is 165 two-repository tasks, five three-repository tasks, and one five-repository task. §1
• Joint necessity is checked by enumerating all 2^N mixed-version configurations and requiring failure in the all-base and every partially patched configuration, with success only when all required patches are applied together. §3.2
• Human screening excludes candidates where additional repository involvement is limited to documentation, release notes, changelogs, or other non-implementation changes. §3.2
• The agent-facing Docker image excludes fixed checkouts, gold patches, hidden tests, and artifacts produced from fixed versions. §3.2
• Failure analysis classifies unsuccessful trajectories into cross-repository inconsistency, incomplete implementation, compatibility regression, logic/state errors, and boundary handling. §5.4
• Manual trajectory inspection finds that GPT-6.1 Sol and Claude Sonnet 5.5 identify relevant code locations and begin modifying source earlier, while DeepSeek V4.1 Flash and GLM-5.3 Flash spend more interactions exploring structures and dependencies. §5.3