From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

What it claims

P³IBench is a 533-sample, 429-video streaming benchmark. A model receives an instruction, watches frames arrive causally, and must choose at each step whether to stay silent or respond. The best evaluated system, GPT-5.6-Luna, reaches only 46.59 PRS-B.

The two model families fail in opposite ways:

• Offline MLLMs adapted to streaming over-respond, producing unnecessary or repeated outputs.
• Native proactive streaming models are too conservative. They miss warranted responses or respond late.

The authors' P³I-Assistant-4B is built on Qwen3-VL-4B with SFT then RL. It scores 42.12 PRS-B, up from 28.62 for Qwen3-VL-4B-Instruct.

How it works

Tasks and sources (5 task groups, 836 ground-truth response targets):

Ground-truth trigger times are spread across the early, middle and late thirds of the videos: 37.8%, 39.7% and 22.5% of events respectively.

Metric. The pairwise score is s = c(ŷ, y) · exp(−0.05·|t − τ|).

• The correctness term c is computed differently by task:
• For structured tasks, c is a rule-based 0/1.
• For the open-ended intervention subset, c is an LLM-judged value in [0,1].
• Each model output is matched to at most one target.
• Two aggregate scores are reported, PRS-U and PRS-B. Both penalize missed, unnecessary and repeated responses.
• Reading timestamps as seconds, a response about 14 s off target loses half its credit.

Evidence

Overall average PRS-B, from Figure 1b:

What this shows:

• An adapted offline model, Qwen3-VL-8B, outscores every native proactive model except the authors' fine-tune.
• MMDuet2 ranks last, even though it was trained specifically for response-or-silence decisions.
• The fine-tune's +13.5 PRS-B gain over its base model brings a 4B model roughly level with Qwen3-VL-8B.

How to use it

The page links to a homepage, code and dataset, but gives no commands, licence or weight details.

The evaluation is a causal loop. At each sampling step the model receives frames V≤t, limited by its context window, plus its own response history. It emits either nothing or a pair (t, ŷ). Offline models must be wrapped in this loop to be evaluated.

Limits

• The fetched text is truncated partway through the metric section. Not included:
• how PRS-U differs from PRS-B;
• the matching rule;
• Table 2 per-task numbers;
• the training-data recipe and size for P³I-Assistant-4B;
• the temporal and streaming-setting ablations.
• Some subsets are small. Object Monitoring and Event Intervention have 50 samples each, so per-task scores there will be noisy.
• Intervention scores rely on an LLM judge, and the page does not name which one.
• Event Monitoring boundaries are largely VLM-generated. They pass through agreement checks and are "inspected or manually refined", but are not fully hand-labelled.
• The authors note that the fine-tune's gains are uneven across capabilities.
• Accident Warning and Crime Alert use only Nexar and UCF-Crime, so risk-prediction coverage is limited to those two domains.