AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

AskStream introduces an ask action and a multi-turn benchmark with Ask-F1 and StreamHarness for evaluating when and what streaming video models should ask under video and user uncertainty.

Prior proactive streaming benchmarks assume the information needed to answer is fully contained in the stream and will eventually become available, so models only wait or answer. The paper argues real-world information is often incomplete, which can cause endless waiting or hallucinated responses. It reports that correct evidence reduces uncertainty and raises accuracy from 42.3% to 62.3%, whereas mismatched evidence provides little benefit.

AskStream models the task as a POMDP with latent missing information and three actions: WAIT, ASK, ANSWER, using only observed video, query, and prior interactions. It divides uncertainty into video uncertainty with explicit and implicit forms and user uncertainty with intent and preference forms, yielding eight subtasks. Events use two annotated temporal patterns: Type I waits before asking and Type II asks before waiting, with frame-level windows for valid ASK and ANSWER. Ask-F1 jointly scores timing and content correctness of ASK and ANSWER. StreamHarness keeps the most recent 10 frames at 1 fps, summarizes every 10 frames into mid-term memory and every four mid-term memories into long-term blocks, retaining up to eight long-term blocks; native streaming models use their own memory. Evaluation covers multiple-choice and open-ended settings, with Gemini-3.8 Flash as the open-ended judge.

• In the multiple-choice main results, Gemini-3.8 Flash is the strongest offline model overall, while the best streaming proactive model AURA scores much lower. §4.2
• Providing oracle information about when to ASK yields the largest recovery on KVEM. §4.3.1
• Open-ended performance drops substantially, Gemini-3.8 Flash has the strongest overall Ask-F1 but its S_ask remains lower than S_ans, and AURA reaches only a low score. §A.4
• Contextual memory has little effect on video uncertainty but improves user uncertainty on AUQ and UPM. §4.3.2
• Increasing the ASK budget from 1 to 16 improves performance but gradually saturates. §4.3.3
• Removing StreamHarness hierarchical memory reduces Ask-F1, and uniform sampling causes larger drops. §A.6

• Each turn allows at most one ASK action, whereas practical scenarios may require repeated clarification.
• AskStream does not consider embodied agents such as vision-language-action models that coordinate perception, information seeking, and physical actions.
• Each interaction is initiated by an explicit user query; autonomous detection of information insufficiency without a direct question is not evaluated.
• Frame-level windows may not cover real-world uncertainty that evolves over longer, more complex horizons with overlapping, discontinuous, or dynamically changing ASK and ANSWER opportunities.

• AskStream extends the proactive wait-and-answer action space with ask and is presented as the first multi-turn benchmark to incorporate an ask action. §1
• Video uncertainty includes explicit subtasks KVEM and VIM and implicit subtasks ICM and PACM. §3.3.1
• User uncertainty includes intent subtasks AUQ and WUQ and preference subtasks UPM and UPC. §3.3.2
• Type I events wait before asking, Type II events ask before waiting, and each turn allows at most one ASK and ANSWER. §3.1
• Ask-F1 jointly evaluates the temporal and content correctness of both ASK and ANSWER. §3.2
• AskStream comprises 438 videos and 1,401 questions, averages 3.2 interaction turns per video, and contains 2714 manually annotated timestamps. §3.5
• StreamHarness retains recent frames and compresses earlier observations through mid-term and long-term memory. §4.1.2
• Early ASK is the main error type across offline models in the multiple-choice setting, and providing oracle information about both when and what to ask improves Ask-F1 by 43.83%. §1