OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

What it claims

OmniCompass tests whether a video model can find a referenced target in time (when), in space (where), and answer about it (what). It also checks whether the model behaves the same when the target is given in text versus text plus a reference image. The headline results are headroom and a split by modality:

• Low ceiling. The best overall OmniCompass-VL scores are 42.8% for the top proprietary model (Gemini-3.8-Flash) and 39.6% for the top open-source model (Kimi-K3).
• Open-source gap is small on vision, large on audio. The best proprietary model beats the best open-source model by 3.2% on the VL track and by 32.8% on the AV track, both in CS Avg. If you are picking an open model for audio-visual work, that second gap matters most.

How it works

• Video durations. Short is 30–90 s, medium is 90–300 s, long is ≥300 s. Clips run from 30 s to 88.4 min. Average length is 354 s on VL and 427.5 s on AV.
• Paired target specifications. Each instance comes in a Text version and a Text+Image version. Both share the same video, task semantics and ground truth. A new metric, the Cross-Specification Consistency Score (CS), scores accuracy and cross-setting consistency together. The page is cut off before CS is formally defined.
• Data.
• 820 videos totalling 80.6 hours, across seven domains and 52 subcategories.
• Public sports, traffic, surveillance and documentary footage is combined with 240 self-collected movie, TV, variety and news videos. Those 240 videos form the AV track and are shared with VL.
• Annotation took about 1,800 human-hours, covering training, calibration, annotation, expert adjudication and revision.
• Position against prior benchmarks. In the authors' comparison table:
• V-STaR covers TG, SG and QA but only with text targets and no audio.
• IV-Bench uses image targets but only for QA.
• Audio-visual benchmarks (WorldSense, Video-MME-v2, JointAVBench and others) have no grounding tasks.

Evidence

• Strong QA can hide weak grounding. High QA scores sit alongside weak grounding and inconsistent predictions across the paired settings. This holds even when aggregate accuracy is similar in both settings.
• Image references rarely help. Text+Image raised the single-setting average for only 11 of 36 models. Supplying a reference image is not a reliable win.
• Timestamps help. Giving timestamps in the input substantially improved performance across tasks and both tracks.
• Audio helps QA and TG. On the AV track, adding audio consistently improved QA and temporal grounding. The page says nothing about audio's effect on SG.
• Weakest model shown. The lowest Figure 1 entry is 15.7, which appears to be LLaVA-Video-7B based on the figure's ordering. The caption names only the top two models.

How to use it

The page links a homepage, a GitHub repo and a Hugging Face release, but the extracted text includes no URLs, loading code, evaluation commands or licence.

Limits

• Truncated page. The fetched text stops at §3.1, so the CS formula, per-task metrics (e.g., IoU thresholds), QA format and per-model tables are missing.
• Partial consistency coverage. The paired-specification analysis covers 36 models, not all 50.
• Small AV track. It has only 240 videos and 720 instances, so per-category numbers there will be thin.
• Unclear provenance and licensing. The self-collected movie, TV and news content raises redistribution questions the page does not address. Check the licence on the release before adopting.