Turns Matter: Advancing Streaming Video RL via Turn-Level Credit

TurnWise applies turn-level remaining-return advantages and turn-mean loss aggregation to streaming video GRPO, improving benchmark scores and response selectivity.

Streaming video models repeatedly decide whether to respond or keep observing, and recent GRPO approaches assign one trajectory-level advantage to every turn. Shared credit can reinforce redundant replies after early successes and suppress useful follow-ups after early failures. Token-mean loss aggregation also weights long responses more than short waiting actions. The paper targets this mismatch between trajectory-level feedback and turn-level interaction.

TurnWise keeps the PAUC-based trajectory reward but refines credit at each decision turn. It scores silence-completed prefixes U_i,t by retaining the first t actions and setting later actions to NO REPLY, then forms remaining returns F_i,t = R_i - U_i,t-1. Group-relative advantages are computed from these remaining returns, with an accrued-reward correction scaled by alpha and clipped to plus or minus a_max; alpha warms up from 0 to 1 over the first 10% of scheduled updates. For policy optimization, turn-mean aggregation averages token losses within each turn and then across turns, giving every response-or-wait decision weight 1/T_i. The unified objective combines corrected turn advantages with denominator B T_i L_i,t, and the prefix computation reuses cached assessments without a learned critic or extra rollouts.

• TurnWise achieves the best aggregate scores on ProactiveVQA, OVO-Timing, and StreamingBench with Qwen2.5-VL-3B, reaching PAUC 44.69, Timing F1 43.01, and StreamingBench Avg 49.43, with aggregate deltas +3.96, +3.87, and +0.69 over the strongest non-TurnWise result. §5.1
• With Qwen3-VL-4B, TurnWise reaches PAUC 42.63, Timing F1 37.73, and StreamingBench Avg 58.96, with aggregate deltas +6.26, +1.92, and +2.38. §5.1
• The component analysis reports the highest task scores and lower repetition for the combination: TurnWise obtains PAUC 43.56, repetition 5.98, Timing F1 21.59, and Avg 49.43, versus credit-only 43.12, 15.53, 18.63, 48.94 and turn-mean-only 41.87, 9.78, 8.17, 47.58. §5.1
• Over the final 50 updates, turn-mean aggregation raises mean reward from 0.657 to 0.711 and reduces repetition-penalty magnitude from 0.069 to 0.036; outside-window reply fraction falls from 0.537 to 0.289 and non-silent decision fraction from 62.2% to 37.0%. §5.2
• Turn-level credit gives negative advantages to 59.83% of repetitive replies that receive positive global advantages (432 of 722); 21.20% of decisions change advantage sign, and 53.15% of trajectories receive both positive and negative turn advantages; across all 2,972 Judge-labeled repetitive replies, the negative-advantage share increases from 75.7% to 81.6%. §5.3

• The paper has no dedicated limitations section; experiments cover only Qwen2.5-VL-3B and Qwen3-VL-4B on ProactiveVQA, OVO-Timing, and StreamingBench, leaving other backbones and streaming tasks untested.
• Training uses a PAUC reward judged by gpt-oss-20b and evaluation uses GPT-5.6-luna; the paper reports no robustness test to judge-model variation.
• Training instances use 20-second target segments and at most 90 frames sampled every two seconds, so longer horizons and different frame budgets are not evaluated.

• At each decision turn the policy generates either a textual response or the explicit waiting output NO REPLY, and waiting is optimized alongside responses. §3.1
• Complete trajectories are scored with a PAUC-based reward that combines response quality and timeliness with repetition, out-of-window, and long-prefix penalties. §3.1
• Standard GRPO converts the trajectory score into one group-relative advantage and broadcasts it to every action token. §3.1
• TurnWise scores silence-completed prefixes and uses the remaining return from each turn to construct turn-specific group-relative advantages. §3.2
• The advantage correction subtracts group-relative accrued reward before the current decision and clips to plus or minus 5, with alpha warmed from 0 to 1 over the first 10% of updates. §3.2
• Turn-mean policy optimization averages token losses within each turn and then across turns, so each response-or-wait decision receives weight 1/T_i. §3.3
• The unified TurnWise objective uses corrected turn advantages and a denominator B T_i L_i,t that makes decision weights independent of response length. §3.4
• Silence-completed prefix scoring reuses cached quality and repetition assessments, requiring no learned critic, additional rollouts, or judge calls. §3.2