OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

What it claims

OmniCapBench is a NeurIPS 2026 Evaluations and Datasets track paper from Tencent Hunyuan, with NTU and Northumbria co-authors. Its central change is to the output format: models must emit structured tracks rather than a free-form caption. The page puts these numbers on it:

• Size: 786 videos totalling 12.8 hours.
• Duration mix: 61.32% are under 1 minute, 29.01% run 1–3 minutes and 9.67% run 3–5 minutes.
• Categories: 20 level-1 and 125 level-2.
• Annotations: 5,818 references (7.40 per video), 11,419 shots (14.53), 39,160 subshots (49.82) and 6,537 audio events (8.32), of which 5,370 are dialogue.

The headline claim is a comparison table against AuroraCap, VCapsBench, UGC-VideoCap, video-SALMONN 2, Omni-Cloze, LongVALE, TimeChat and OmniScript. OmniCapBench is the only one marked as natively supporting all four of identity tracking, temporal grounding, audio-visual association and diagnostic traceability.

How it works

The schema extends the Multi-Stream Scene Script (MTSS). It splits shots into finer subshots and replaces point timestamps with continuous time ranges. It has three tracks:

• References: persistent IDs for people, objects and scenes.
• Events: audio and dialogue with time bounds, for example "dog bark 10.5s–12.0s".
• Shots: visual segments that cross-link to references and events.

Scoring has two stages. First, deterministic rules plus a bounded LLM matcher check IDs, temporal bounds and cross-links. Then localized LLMs judge only the semantic equivalence of the aligned fields.

Ground truth is built in three stages:

1. Filtering for audio presence, duration, quality and density.
2. Track generation in dependency order (R→E→H), using a multi-model loop.
3. Audits: programmatic validators plus targeted human review of boundary and semantic disagreements.

flowchart LR
  G[Generator proposes tracks] --> V{Evaluator ensemble + schema validator}
  V -- invalid --> G
  V -- unfixable --> D[Discard]
  V -- valid --> R[Local refine text only<br/>IDs/times/links frozen]
  R --> S[Selector picks best]
  S --> A[Programmatic + human audit]

During refinement, only the descriptions are enriched. IDs, time ranges and links stay fixed so the text edits cannot corrupt verified structure. Annotation frames carry a red timestamp overlay, but evaluated models receive clean frames.

Evidence

The paper describes three experiments:

• Capability results split into deterministic-rule scores (Table 3) and semantic scores (Table 4).
• An audit of holistic scoring across models, metrics and formats (Table B.7), which the authors say masks local errors and rewards structurally flawed outputs.
• A weak-vs-strong judge comparison (Table 5), which they say shows global LLM scoring is unstable. They also report that QA probes can mark correct captions wrong.

None of these tables' numbers appear in the fetched text.

How to use it

The page gives a project site, https://01yzzyu.github.io/OmniCapBench/. It states no code, download, licence or run command.

Models under test have to produce the three-track structured output. Free-form captioners cannot be scored directly.

Limits

• No results available: the fetched text stops in the data-construction section. The evaluated models, their scores and the judge LLMs used are not available here.
• Model-built ground truth: references are generated by a multi-model loop and only partly human-audited ("targeted"). The paper does not name the generator, evaluator or selector models.
• Unclear data rights: some videos come from public media platforms, and the page states no redistribution licence.
• Possible format confound: the page does not show how much scores depend on a model's ability to follow the structured schema rather than on its perception. Check this before comparing models.
• Short videos: all videos are at most 5 minutes, and most are under 1 minute, so "long-horizon" here means minutes.