StreamDuo: A Live Commentator and a Research Agent for Unseen Games

StreamDuo pairs a trainable streaming commentator with a frozen wiki-reading research agent, trained on one game, to improve entity naming and reduce source-game leakage on ten unseen games.

Streaming video LLMs can commentate live gameplay, but they are trained and tested on the same games. In cross-game streaming commentary, current commentators transfer their timing but not their knowledge of the game world. When retrained on Baldur’s Gate 3 alone, a Proact-VL commentator rarely names what is on screen in ten unseen games and sometimes reaches for training-game names.

StreamDuo uses a commentator agent that watches each second of the stream and decides whether to speak, paired with a frozen research agent that sees only the frames sent when the commentator asks. The research agent reads the played game’s wiki and returns a few entries two seconds later while the stream continues. Only the commentator is trained, on a single source game, in two stages. Stage 1 uses information-gain labels to teach when to ask, and Stage 2 uses on-policy self-distillation from a frozen copy of the commentator that sees an ideal answer. At test time the teacher and ideal answer are not used; the commentator asks with its frames and speaks with whatever the research agent returns.

• Compared with Proact-VL trained from the same base on the same clips, StreamDuo raises entity recall and K-strict. §4.2
• StreamDuo cuts source-game leakage while naming more entities than Proact-VL. §4.8
• OPSD gives the largest component gain, and the three components together raise Ent-R and K-strict over the Proact-VL recipe. §4.3
• Information-gain ask labels outperform naive labels on Ent-R and FinalQ. §4.4
• Asking every 3 s nearly doubles recall but lowers Ent-F1 and judge scores, while the learned policy asks 264 times over 2,470 clips and gets the highest Ent-F1. §4.4
• After an answer arrives two seconds later, entity naming jumps to 32%, 4.6 times that of a random second; with asking disabled, the same seconds name an entity only 5% of the time. §4.5

• The paper trains only on a single source game, Baldur’s Gate 3, and tests on ten unseen games; it does not report training on multiple source games or other source-game choices.
• The research-agent swap is evaluated only within the Qwen3-VL family and the same BM25 wiki search, so other agent families or retrieval methods are not tested.
• Commentary quality is judged by an LLM; the paper reports no human evaluation, leaving human-perceived quality open.
• Qualitative examples show the commentator does not always filter answers well; in StarCraft II it repeats Warden Support Alpha, an unrelated entry, and it sometimes reads wiki entries instead of commenting.

• Cross-game streaming commentary trains on one source game and tests on unseen games; timing transfers across games, but knowledge does not. §1
• When the commentator sees something it cannot name, it sends the current frames to a frozen research agent, which returns a few wiki entries two seconds later while the stream continues. §3.2
• Stage 1 labels second a minus delta as an ask moment when the information gain g_a exceeds gamma, so answers are requested only where they would help reproduce the human caster’s line. §3.3
• Entities of the source game are renamed consistently and at random during training so that a name is evidence only within its own clip. §3.1
• Stage 2 uses on-policy self-distillation from a frozen copy of the commentator that sees an ideal answer, with reverse KL over student samples and amplification; only low-rank adapter weights are updated. §3.4
• The learned ask policy is rare: 264 questions over 2,470 clips, and before a question the commentator names an entity in only 1 to 4% of seconds, below the 7% of a random second. §4.5
• The research agent answers in 0.41 s on average and at most 0.71 s over 200 test moments; the commentator needs 0.26 s of compute per second of video and at most 0.33 s on 20 test clips. §6.1
• When the research agent is swapped, Ent-R stays between 0.101 and 0.116 and judge scores stay within 0.03, and 30B-A3B does not improve on 8B. §4.7