TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

What it claims

TRIAD is a diagnostic benchmark from Tencent Hunyuan and Peking University, accepted to the NeurIPS 2026 Evaluations and Datasets track. It tests whether omni-modal models can combine three individually insufficient cues (text, image, audio) into one answer. The best of 14 evaluated systems reaches 90.4 query accuracy but only 71.2 group accuracy under multiple choice, against a human baseline of 91.6. The authors read this gap as evidence that per-item success does not imply consistent scene-level disambiguation. The page text shown does not name which model scored best.

How it works

Each query is a tuple of text context, image, audio clip, question and option set. An item is well posed only if the full triplet leaves exactly one consistent option and every two-modality view leaves more than one. A modality can be clear on its own and still count, because it eliminates an option the other two leave open.

Queries sit in scene groups of 2, 3, 4 or 8 "sister" queries. Sisters share a scenario and answer space but perturb one modality, so the correct answer changes with that cue. Group accuracy (Acc_g), the primary metric, credits a group only if every sister is correct. This penalizes answering from a fixed scene prior.

The taxonomy has six leaf categories per modality:

• Text: anaphoric, syntactic, logical, pragmatic, temporal, lexical.
• Image: semiotic, grounding, spatial, kinematic, observational, contextual.
• Audio: referential, prosodic, indexical, phonetic, environmental, symbolic.

Every item carries one label from each modality.

Evidence

• Composition: 184 English and 143 Chinese queries. Option sets have 2 to 8 choices.
• Ambiguity tiers: 230 items have one internally ambiguous modality, 59 have two, 38 have three.
• Evaluation grid: three prompt formats and eight modality conditions, with human and random baselines.
• Counterfactuals: audio is replaced with a transcript and the image with a description.
• Open-ended results: with options removed, every model falls to low single-digit group accuracy. Humans stay substantially more robust.

How to use it

The authors recommend reporting Acc_g as primary, Acc_q as secondary, and open-ended scores as a lower bound on free-response disambiguation. Per-item tags for modality, category, tier, scene group and language support controlled error analysis. A Python reference checker for well-posedness ships in the supplementary material for anyone extending the set. The page gives no repository URL, install command or licence.

Limits

• Scope: the authors call TRIAD diagnostic only. It is not a training set, not a recognition or transcription benchmark, and not a replacement for broad multimodal evaluation.
• Size: at 327 queries and 86 groups, each group moves Acc_g by about 1.2 points.
• Human-judged gate: five annotators (three NLP, two linguistics) review well-posedness with a subset checklist, not an automated check. About 40 percent of candidate items were admitted.
• Synthetic media: a small fraction of images are AI-generated, used only when no real photograph fit. These are documented so users can filter them. Some audio is text-to-speech with human-edited prosody.
• Truncation: the fetched text ends partway through Section 4. Per-model results, inter-annotator agreement figures, and category and tier breakdowns are not visible.