On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

The paper splits training-inference mismatch into an approximately fixed numerical discrepancy and a self-induced policy lag, then proposes DLC, which centers the discrepancy, clips the lag ratio, and resynchronizes the sampler when the lag ratio's effective sample size falls below one half.

On-policy reinforcement learning for large language models assumes training responses are sampled from the current policy, but an inference engine samples responses while a separate trainer computes gradients, making the data off-policy. The mismatch has two sources, engine numerics and stale weights from asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. The paper asks which property of a mismatch source decides the correction it needs.

The authors evaluate the trainer at the sampler's weights to split each sampled-token error exactly into a numerical discrepancy of the engine and a policy lag of the trainer. They follow both terms through training on Qwen3-0.6B and Qwen3-1.7B on Countdown with rewards ±1, 64 prompts with eight samples per step, SGD at learning rate 10^{-2}, and three runs of 40 steps per configuration. They compare reference, quantized, stale, and combined samplers, and derive the residual bias left by score centering and importance sampling. They analyze the policy lag as a feedback loop in which the residual bias enters the next update and enlarges the next lag. From this account they derive the discrepancy-lag correction, which centers the numerical discrepancy, clips the lag ratio against the sampler's weights, and resynchronizes the sampler when the effective sample size of the lag ratio falls below one half.

• Under quantization, importance sampling and score centering, alone or combined, stay at the noise floor while the policy gradient reaches 13 times the floor, and score centering matches the 46.5% test accuracy of on-policy training. §4.1 Source dependence of the corrections
• Under staleness, score centering reaches 32 times the floor and ends at 12.2%, while importance sampling with centering stays at 2.1 times the floor and ends at 32.0%. §4.1 Source dependence of the corrections
• The error of the quantized sampler is numerical discrepancy of about 0.02 nats per token throughout training, and the stale error is policy lag with numerical discrepancy below 10^{-3}; the magnitude of the stale sequence offset grows by about 0.7 nats per step. §4.2 Exact split of the sampled-token error
• Under score centering the measured residual bias grows linearly in the accumulated update with R^2 of 0.95 to 0.99 per run and reaches 2.5 times the gradient noise by step 40, whereas under importance sampling it stays within the noise; the fit of the excess update norm gives c = 0.52 at R^2 = 0.987 against 0.90 for the linear term alone. §5.3 Loop gain
• On Qwen3-1.7B the measured effective sample size of importance sampling falls below one half at step 11 at a mean lag of -0.15 nats per token, three steps before the Gaussian value at step 14, and the deviation of importance sampling grows to 152 times the floor by step 30. §6.1 Tail of the ratio
• On Qwen3-1.7B under staleness, DLC stays within 1.2 times the floor through step 40 and ends at 0.6 with 51.9% test accuracy, against 57.6% for on-policy training, while importance sampling with resynchronization ends at 2.6 times the floor and 36.6% test accuracy. §7.2 Deviation and test accuracy

• The main trainings are short, 40 steps, and replay of the sampler on shared checkpoints keeps the measurements on trajectories that have not collapsed, so longer-horizon behavior is not established (stated in Section 3.2).
• DLC adds an extra trainer forward pass at the sampler's weights and requires the sampler's top-128 log-probabilities; the paper reports 113.2 seconds per step for DLC against 53.2 for importance sampling on Qwen3-1.7B under a stale sampler (stated in Appendix C).
• The residual-bias prediction leaves out the continuation term and uses a linearized prefix term; with exact weights the norm ratio falls to between 1.9 and 7.9 but the cosine stays modest (stated in Sections 5.2 and F).
• The Qwen3-8B and continuous-asynchrony comparisons use one run per correction, so run-to-run variability on those settings is left open by the experiments (stated in Appendix C).

• After infrastructure alignment, only the weight version and engine numerics reach the algorithm; the weight version gives the policy lag, which grows with the update, and the engine numerics give the numerical discrepancy, which stays approximately fixed in training. §2.1 Training-inference mismatch
• Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into the policy lag and the numerical discrepancy. §4.2 Exact split of the sampled-token error
• The numerical discrepancy does not depend on the accumulated update, so centering removes its first-order bias, whereas the policy lag is self-induced and grows only because the trainer moves. §4.2 Exact split of the sampled-token error
• The loop depends on learning rate and step count only through eta k, and the excess update norm of score centering over importance sampling carries a positive quadratic coefficient in each of the six fits. §5.3 Loop gain
• The validity range of ratio weighting ends earlier than the Gaussian prediction, at a mean lag of -0.15 nats per token on Qwen3-1.7B, because positive cumulants of the lag lower the effective sample size. §6.1 Tail of the ratio
• The PPO reference policy determines which mismatch term enters the clipped ratio: with the sampler's weights as reference the policy lag lies inside the ratio and the numerical discrepancy outside it, while with the trainer before each batch both stay outside the ratio. §6.2 Reference policy of PPO
• DLC assigns the numerical discrepancy to centering, the lag token term to clipping of the lag ratio against the sampler's weights, and the unreachable lag to sampler resynchronization when the effective sample size falls below one half. §7.1 Method
• DLC without resynchronization collapses while its deviation on the sampler's prefixes stays at the floor; the off-support share of the trainer's own responses reaches about 90% at the collapses and ranks the drop below theta_0 at a Spearman correlation of 0.80. §7.3 Mass off the support of the initial policy