DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

What it claims

DIVE is a 525-instance audio/video benchmark from Huazhong University of Science and Technology and Princeton (arXiv, October 2026). It scores full-duplex models on three separate things: whether they speak or stay silent, when they respond, and what they say. The authors argue that existing streaming benchmarks mostly use timestamped question answering. They also argue that existing duplex benchmarks judge isolated response windows rather than sustained interaction. Their Table 1 marks DIVE as the only listed benchmark covering all six of its task types. The closest prior benchmarks it names are OmniInteract, Omni-DuplexEval and VideoFDB.

How it works

The 525 instances come from 413 base scenarios. Each task draws on its own sources:

• Video Timely Intervention (164): egocentric procedural-error clips from PIE-V, MATT, Ego-EXTRA and CaptainCook4D. The model gets a standing instruction to flag mistakes. Annotators marked the earliest frame at which each mistake is visible.
• Speech Proactive Intervention (50): unsafe situations from LLaVAShield and LlamaPIE, plus hand-written cases. These are recast as spoken narratives, with videos generated by Wan 3.0 where a visual cue is needed.
• Continuous Query Interaction (50): query sequences from Live-Aid plus manual cases with Wan 3.0 video. Visually grounded questions alternate with general-knowledge ones.
• Continuous Action Tracking (37): RepCount, Multi-RepCount and CPR-rac. The model must speak once per completed repetition and stay silent between repetitions.
• Overlapping Response Adaptation (112): interruptions adapted from EchoChain and HumDial-FDBench. They are rewritten as corrections or new constraints that arrive while the model is speaking. There are 56 scenarios, each in two conditions.
• Audio-Visual Addressee Tracking (112): multi-party dialogues from Speak or Stay Silent and When2Speak with added visual cues, plus real videos from Multi-TPC and EasyCom. The model uses gaze and speaker orientation to decide whether it is being addressed. There are 56 scenarios, each in two conditions.

Scoring combines three components. The decision score checks whether the model spoke when required and stayed silent otherwise. The temporal score labels each required response as early, on-time, late or missed against an annotated window. The quality score uses an MLLM-as-judge and grades only the first response assigned to each event.

flowchart TD
  E[Interaction slot] --> D{Should model speak?}
  D -->|Silent required| S[Decision score: penalise any output]
  D -->|Response required| T{Response vs window}
  T --> Early
  T --> OnTime[On time]
  T --> Late
  T --> Missed
  OnTime --> Q[MLLM judge: first response quality]
  Early --> Q
  Late --> Q
  S --> O[Overall score]
  Q --> O

Evidence

The fetched text gives only aggregate results. Leading real-time models reach sub-second average latency. Their average window response rates are at or below 53.77%. The authors single out speech-driven proactive intervention as the weakest area for response rate. Revising an ongoing response after new information arrives is the hardest case for content quality. The fetched text does not include per-model or per-task tables.

Limits

The fetched text breaks off partway through the evaluation section. It does not name the models evaluated, the latency definitions or the judge model. It gives no code, data release or licence. Much of the video is synthetic, generated with Wan 3.0 for speech-intervention, query and interruption cases. Scores on those tasks may differ from real footage. Three tasks are small: SPI and CQI have 50 instances each and CAT has 37. Quality depends on an MLLM judge whose identity is not given in the fetched portion.