AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

What it claims

AskStream shows a large gap between offline and streaming proactive models when the agent must ask for missing information. On Ask-F1, the strongest offline model reaches 52.18%. The best streaming proactive model reaches 7.22%. The authors name two common error types: asking too early (Early ASK) and asking the wrong thing (wrong ASK). Telling models both when and what to ask (an oracle) raises Ask-F1 by 43.83 points, which the authors read as evidence that proactive asking is the main bottleneck.

A pilot study on blurred inputs motivates the ask action. Accuracy is 42.3% on blurred video, 62.3% after asking for and receiving the key evidence, and 63.1% with fully clear input. Mismatched evidence gives little benefit. The model used for this study is described in Appendix A.3, which is not in the fetched text.

How it works

Each query opens an event lasting until the next query. The model picks WAIT, ASK or ANSWER once per second, with at most one ASK and one ANSWER per turn. A correct ASK makes a simulated user supply the missing information, such as a clear video, a switched viewpoint, a photo of a named person or a preference.

Events come in two annotated types:

flowchart TD
  Q[Query arrives] --> T{Uncertainty visible at start?}
  T -->|No: Type I| W1[WAIT until evidence emerges]
  W1 --> A1[ASK inside answer window]
  T -->|Yes: Type II| A2[ASK inside ask window]
  A2 --> W2[WAIT for evidence]
  A1 --> R[ANSWER inside answer window]
  W2 --> R
  R --> S{Ask and answer both correct and on time?}
  S -->|Yes| OK[Turn succeeds]
  S -->|No| F[Turn fails, next query]

Ask-F1 is the harmonic mean of an ask score and an answer score. Each correct action earns a timing credit that decays linearly from 1 at the window start to 0 at its end. The ask score averages over all turns. The answer score averages only over turns with a correct, in-window ask, so answering well cannot compensate for failing to ask.

The tasks split into video uncertainty and user uncertainty, with eight subtasks:

• Video, explicit: key visual evidence missing (blurred entity) and viewpoint information missing.
• Video, implicit: individual concept missing and personalized action concept missing.
• User: ambiguous question, wrong question, preference missing and preference conflict.

Videos come from reused datasets, Blender synthesis and YouTube. They are filtered for length over 3 minutes and resolution of at least 720p. Evidence is then obfuscated by blurring, viewpoint switching, unclear or erroneous queries, and preference edits. Options are model-generated and human-verified, including the time windows.

StreamHarness is the evaluation framework. It keeps recent frames and compresses earlier observations into a hierarchical memory, so offline models can be run in a streaming setting.

Evidence

Table 1 compares AskStream with nine benchmarks. AskStream is the only one with frame-level annotation and an ask action. It averages 9.44 answer options per multiple-choice item, has 8 tasks and an average video duration of 11.0. Of the prior proactive benchmarks, ProactiveVideoQA and ProReady-QA also use time windows.

How to use it

The page links to code, data and a website, but the fetched text does not include the URLs, a licence or run commands.

Limits

The fetched text stops partway through the task taxonomy. Per-model results, the multiple-choice versus open-ended breakdown, model names and the metric analysis in Appendix A.2 are not available here. The main setting allows one ask per turn, so multi-step clarification is not evaluated. User replies are scripted responses to a correct ask rather than live interaction.