What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?

The paper finds that multi-turn agentic RL mainly improves long-horizon execution and error recovery rather than uniformly improving per-step action quality, and its gains depend on sufficient interaction budget.

The paper asks what capabilities multi-turn agentic RL actually improves in LLM agents, since existing evaluations mostly judge success at the trajectory level. It argues that interactive agents have an interaction-depth axis in addition to sampling breadth, and that trajectory success may mask budget dependence or step-level quality changes. The authors aim to test whether RL gains hold under varying interaction budgets and whether single-step actions improve.

The authors formalize Pass@(K,T), combining sampling breadth K with maximum interaction turns T, and sweep Pass@K and Pass@T at fixed k=8. They primarily train with GRPO, plus GiGPO and PPO, using Qwen3-4B and Qwen3-8B on DeepCoder-MT, BFCL, InterCode-SQL, and FrozenLake, with additional Terminal-Bench results in the appendix. They compare matched-state step-level action quality by sampling G=16 actions from base and RL policies, using exact Q* values for FrozenLake and unit-test pass rates for DeepCoder-MT, and they construct error states. They analyze action NLL under the base policy and cumulative log-likelihood ratio, and track trajectory-level versus step-level rewards during training. They also discuss process supervision signals and introduce BudgetMix RL, which samples interaction budgets at 10% intervals during training.

• As sampling breadth k increases, the RL model consistently outperforms the base model, suggesting multi-turn agentic RL can push trajectory-level success beyond the base model's empirical Pass@K boundary. §3.1
• With sampling breadth fixed at k=8, the RL advantage depends on a sufficient interaction budget: it becomes more pronounced as T increases and can diminish or reverse under tight budgets, while structural upper bounds show headroom remains at small T. §3.2
• Agentic RL improves step-level decision quality on only 46.8% of DeepCoder-MT states and 42.8% of FrozenLake states, but on error states RL selects the superior action in 65.7% of FrozenLake cases and 67.3% of DeepCoder-MT cases; continuation success is 51.3% vs. 38.6% on FrozenLake and 38.5% vs. 22.9% on DeepCoder-MT. §3.3
• Under the base policy, mean action NLLs for RL and base actions are 12.20 and 10.25, while the cumulative LLR of successful RL trajectories reaches approximately 228 compared with 26 for successful base trajectories. §4.1
• On DeepCoder-MT, during standard agentic RL training the trajectory-level reward R_multi rises by 0.331 while the per-step reward R_single drops by 0.097. §4.2
• BudgetMix outperforms full-budget RL under tight budgets at 25%, 50%, and 75% Tmax with average performance gains of 7.5, 5.3, and 4.0 percentage points, respectively, and maintains parity at 100% Tmax. §5.4

• Step-level action-quality evaluation is restricted to tasks with reliable step-level measures, such as FrozenLake and DeepCoder-MT, because reliable process rewards remain difficult to obtain in agentic settings.
• The error-state constructions are proxies rather than rigorous definitions of error states.
• High-quality process supervision can mitigate budget reliance, but the paper states that precise process supervision is rarely accessible in realistic agent tasks; on BFCL, dense signal methods fail to improve constrained-budget performance.
• BudgetMix is presented as a preliminary discussion of candidate solutions rather than a fully established solution.

• The paper introduces Pass@(K,T) to separate sampling breadth K from interaction depth T; Pass@T fixes the sampling budget at a constant C. §2.2
• RL models consistently outperform base models as the number of sampled trajectories k increases, suggesting agentic RL expands the empirical Pass@K capability boundary. §3.1
• The RL advantage depends on sufficient interaction budget and can diminish or reverse under tight budgets; structural upper bounds show substantial headroom remains at small T. §3.2
• Step-level quality is not uniformly improved; the RL advantage is more pronounced at error states and later in trajectories, with normalized depth d_t=t/Tmax. §3.3
• Likelihood analysis shows RL actions remain locally plausible under the base policy, while complete RL trajectories receive much higher likelihood under the RL policy than under the base policy. §4.1
• During standard RL training, trajectory-level reward rises while per-step reward does not improve and can decline. §4.2
• Oracle Q-value process rewards and GiGPO stabilize FrozenLake across budgets, but on BFCL no exact step-level oracle is available and dense signal methods such as GiGPO and PPO fail to enhance performance under constrained budgets. §5.1
• BudgetMix turns the interaction budget into a training dimension by sampling budgets at 10% intervals for each task instance. §5.2