StreamClarify: Proactive Clarification for Grounded Streaming Video Question Answering

What it claims

StreamClarify adds a clarify-or-answer gate and two-tier memory to a frozen streaming VLM, with no training. The abstract reports a gain of up to 8.2 Macro accuracy points over backbone-matched baselines on three backbones. Two further numbers isolate the parts. At comparable trigger rates, entropy-based triggering is more than 18 points more precise than random triggering at flagging answers that were initially wrong. With oracle user guidance, clarification adds up to 3.5 Macro accuracy points. A separate oracle-assisted simulation on Qwen3-VL uses generated clarifying questions and referent-only user replies on Ambiguous QA, and the entropy-gated condition improves accuracy there by 8.3 points.

How it works

flowchart TD
  F[Incoming frame] --> TP[TP-KV active cache]
  F --> AR[CPU archive: vision tokens + SigLIP2 key]
  Q[User query] --> S1[Stage 1: Yes/No evidence judgment]
  TP --> S1
  S1 -->|entropy at or below threshold| S2[Stage 2: answer from TP-KV]
  S1 -->|entropy above threshold| C[Request clarification]
  C --> R[Guidance to search phrases]
  R --> AR
  AR -->|top-K clips prefilled| S2b[Answer from TP-KV + retrieved clips]

Stage 1 asks the model whether its answer would rely on observed video rather than general knowledge, and it emits [Yes] or [No]. The gate is the binary entropy H of p([Yes]) at the label token. If H exceeds a fixed per-backbone threshold, set from the no-clarification distribution, answering is suspended. Stage 2 shares the same KV cache, so the check costs one short generation and needs no sampling or repeated forward passes.

TP-KV (temporal-pyramid KV cache) keeps a recent window at stride 1. When the cache exceeds the token budget L_max, older tokens are stride-sampled at progressively coarser pyramid levels. The oldest tokens are evicted if the compressed history still exceeds the budget set by compression ratio ρ. The kept tokens are then re-prefilled in chronological order. Direct answers use only TP-KV. The archive stores backbone vision tokens plus a SigLIP2 image-text key for every frame, and only clarified queries use it.

Evidence

StreamClarify-QA has 1,837 QA pairs, evaluated causally at each question's timestamp. StreamBench contributes 1,556 deduplicated open-ended pairs from 260 videos (average 253 s). OVO-Bench contributes 281 multiple-choice pairs from 95 videos (average 383 s). The OVO-Bench videos are those with at least one Hallucination Detection question, and questions less than 5 s after the previous one are dropped.

There are two constructed splits:

• Referential Ambiguity (302 pairs): Qwen3-14B swaps a noun phrase for a pronoun when captions show several plausible referents. Human review cut the 404 candidates to 302.
• Unanswerable (540 pairs): 389 StreamBench items were turned into multiple choice with caption-checked distractors and "None of the above", plus 151 unchanged OVO-Bench HLD questions. Three rounds of Gemini screening on the raw video left seven cases, and human review confirmed all seven labels.

The dense captions come from MiniCPM-V 4.5. The extracted text ends before the results tables, so per-backbone and per-split numbers are not available here.

How to use it

The method is training-free and wraps a frozen backbone. It needs a per-backbone entropy threshold, the token budget L_max, the ratio ρ, the retrieval depth K and SigLIP2 for archive keys. Prompt templates are in Supplementary Sec. G, and the update procedure is Supplementary Algorithm 1. The project page is zhuchens-uiuc.github.io/StreamClarify-page/. The text gives no code release, licence or install details.

Limits

• The 3.5-point clarification gain and the 8.3-point simulation both assume oracle user guidance. The core framework does not include a clarifying-question generator.
• The archive grows with stream length. A retention window is the suggested bound, and archive growth is measured only on an approximately 30-minute stream (Supplementary Sec. F.2).
• Prior dialogue turns are not a benchmark input. Clarification applies only to the current query.
• Only Qwen3-VL is named among the three backbones in the available text.