R^2Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

R2Orch nests self-play orchestration with a learnability-based Proposer and randomized skill-withdrawal tests, improving a Director toward zero-skill deployment across twelve benchmarks.

Existing orchestration learning relies on hand-designed curricula, suffers advantage collapse when a task's rollouts all reach the same outcome, and uses unreliable skill evaluations based on one noisy score. Self-play with skills in context can also collapse advantages once skills are withdrawn. The paper targets a Director that succeeds with no skills in context while choosing tasks and withdrawing skills jointly.

R2Orch has an inner loop in which a Proposer chooses tasks and a Director builds agent graphs, and an outer loop that withdraws skills. Recursive Learnability rewards the Proposer for tasks whose rollouts keep differing with the candidate skill withheld, using the curvature 4p^-(1-p^-). Skill Withdrawal randomly hides a candidate skill in part of each rollout group, yielding an unbiased paired contrast and a Bayesian posterior for withdrawal and admission decisions. The loops are coupled by a randomized context q_x and two time scales. The paper states that zero-skill success equals starting success plus inner gains minus outer costs.

• R2Orch is best on all twelve benchmarks and its lead holds on the held-out half at zero skills. §5
• R2Orch reports average IID score 89.79 and average OOD score 78.79. §5
• The Director trained with Qwen3.5-9B Workers improves all six other backbones without retraining and narrows the IID range across backbones from 25.7 to 6.4 points. §5
• All 12 withdrawals pass Eq. (14) on the card's own contrasts, with posterior E[Delta+] at most 0.019, and seed cards no longer raise held-out accuracy around them. §5
• New groups on tasks of intermediate success rate reach mixed outcomes about four times as often as at p=0 or p=1. §5

• R2Orch starts from initial skills hand-customized for each dataset, and deriving initial skills automatically for a new domain remains an open problem (stated in Limitations).
• The paper evaluates on public benchmarks and six additional Worker backbones without retraining; broader deployment domains and backbones are not tested (experiments leave this open).
• The study reports no human-participant or real-world deployment evaluation; misuse risk is discussed in Ethical considerations but not empirically evaluated (experiments leave this open).

• The inner loop improves the Director on Proposer-chosen tasks, while the outer loop withdraws skills that stop helping. §4.1
• Each rollout group is randomized: most rollouts read K_x without the candidate skill and the visible part reads K_x. §4.1
• Recursive Learnability is 4p^-(1-p^-), vanishes at p^- in {0,1}, and peaks at one half. §4.2
• The Proposer reward uses the hidden part, discounts when only the visible part progresses, and penalizes collapse. §4.2
• The difference of visible and hidden part means is an unbiased paired contrast for a skill's effect. §4.3
• Withdrawal requires n_k at least n_min and posterior E[Delta+] at most epsilon; admission requires q_k at most alpha. §4.3
• Round accounting gives zero-skill success as starting success plus inner gains minus outer costs. §4.1
• The configuration uses G=5 rollouts with 3 hidden and 2 visible, n_min=12, and re-tests withdrawn skills on 30% of tasks. §C