On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation

The paper finds that on-policy and multi-teacher on-policy distillation can match teachers in-distribution while leaving out-of-distribution generalization below the base model or best teachers, and it studies bottlenecks, update geometry, and mitigation methods.

Prior work reports near-perfect in-distribution (IND) distillation performance for multi-teacher on-policy distillation (MOPD), but the paper finds this can mask incomplete capability consolidation beyond the supervised distribution. A preliminary synthetic reasoning study shows that OPD does not fully transfer teachers' generalization capabilities. The paper investigates whether such gaps appear on real-world tasks.

The authors evaluate extrapolative and compositional generalization using Qwen3 models across mathematical reasoning, coding, agent, and long-context capabilities. They compare OPD and MOPD IND and OOD scores against teachers and base models. They examine teacher capability and data coverage as potential bottlenecks and test joint alleviation of these bottlenecks. They analyze update geometry relative to reinforcement learning baselines. They propose geometry-based patching and base-preserving replay to improve OOD performance while largely preserving IND performance.

• Across extrapolative settings, OPD and MOPD reach 96.04-102.73% of their teachers' IND scores. §Abstract
• Across extrapolative settings, OPD and MOPD OOD scores remain 2.45-16.25% below the base model's. §Abstract
• Across three compositional settings, MOPD improves IND performance over the base model, but performance on composed tasks as OOD remains 18.19-24.34% below that of the best teachers. §Abstract
• Jointly alleviating teacher capability and data coverage bottlenecks eliminates OOD forgetting in the experiments. §Abstract
• (M)OPD reduces OOD forgetting on average relative to its reinforcement learning baselines. §Abstract
• Update geometry analysis reveals smaller and more spectrally concentrated updates on average than RL baselines, along with greater shared update structure under MOPD. §Abstract

• Experiments use Qwen3 models only, so generalization to other model families is left open.
• Joint alleviation of bottlenecks eliminates OOD forgetting only in the paper's experiments, leaving broader conditions open.
• The preliminary study is on synthetic reasoning, and the abstract does not establish that the same preliminary finding holds across all real-world settings.

• MOPD extends on-policy distillation to consolidate capabilities from multiple teachers. §Abstract
• Near-perfect IND distillation performance can mask incomplete capability consolidation beyond the supervised distribution. §Abstract
• A preliminary study on synthetic reasoning shows OPD does not fully transfer teachers' generalization capabilities. §Abstract
• The investigation evaluates extrapolative and compositional generalization across mathematical reasoning, coding, agent, and long-context capabilities using Qwen3 models. §Abstract
• Teacher capability and data coverage are examined as two potential bottlenecks. §Abstract
• Geometry-based patching and base-preserving replay are proposed to improve OOD performance while largely preserving IND performance. §Abstract
• The findings identify a blind spot in capability consolidation via (M)OPD and highlight the need for algorithms that explicitly support OOD retention and generalization. §Abstract