Grounding Generative Verification in 3D Content

What it claims

3DVerifierBench has 2,581 human-reviewed examples across ten 3D verification tasks. Each task asks a model to accept or reject an explicit claim about 3D evidence and to explain any rejection. The failures are concentrated in a few tasks, which the summary's single 21.6-point gap hides. In the paper's Figure 1, GPT-6 Astra scores 50.0% on Fine-grained Geometry and 50.0% on Motion Understanding, which equals the random baseline. It scores 54.8% on Identity Preservation, against 92.9% for humans. Its best task is Part Grounding, at 85.0% against 92.5% for humans. Overall, humans score 87.8% and Astra 66.2%. The authors also report that a correct rejection often comes with an explanation that misses the actual contradiction.

How it works

Interface. The input is a task instruction, a claim, the candidate's evidence and optional reference evidence. The model returns True or False plus an explanation. The candidate can be an asset, a part, a grasp or an animation. The benchmark checks whether stated requirements are met, not general plausibility or visual quality.

Metrics. Each task is scored with balanced accuracy, and the ten tasks are weighted equally.

• ExpBA replaces accuracy on negatives with joint answer-and-explanation success. A "reference-semantic Judge" grades the explanations. It sees the task, the claim and both explanations, but no visual evidence and no model identity.
• Linked counterfactuals group a positive anchor with negatives:
• Spec-CF changes the claim and keeps the evidence fixed.
• Cand-CF changes the evidence and keeps the claim fixed.
• A pair or triplet counts only if every answer in it is correct.

Data construction uses three strategies, chosen by task:

flowchart LR
  A[3DGen-Bench, Hi3DBench, DB-3DME, PartNeXt, GAPartNet, ACRONYM] --> CF{Controlled counterfactuals}
  CF --> S[Spec-CF: claim changed]
  CF --> C[Cand-CF: 3D edit]
  F[FaceScape bilinear model] --> ID[Identity pairs]
  T[T-LESS + Laplacian diffusion / AnimalML3D + AnimateAnyMesh] --> EX[Expert-guided FineGeo, Motion]
  S --> R[Expert review]
  C --> R
  ID --> R
  EX --> R
  R --> B[Benchmark 2,581]
  R --> TR[Training 14,126]

• Seven tasks (Object, Part, Material, Part Grounding, Relation, Affordance Region, Grasp) use controlled counterfactuals. Gemini 3.1 Flash filters the source data. Native annotations take precedence over synthetic edits when both exist. Two experts spot-check 10% of the output at each model-assisted review stage.
• Identity has 200 FaceScape face pairs, labelled by whether the identity coefficients changed. Six experts set the thresholds for a perceptible identity change. Expression and texture vary across four nuisance conditions.
• Fine-grained Geometry applies Laplacian perturbations to T-LESS CAD models, with experts choosing the regions. Models are compared against fixed reference photographs.
• Motion pairs AnimalML3D reference motions with AnimateAnyMesh outputs. Six experts guide each of these two tasks.

Evidence

Human and Astra scores are percentages from the human study, where random scores 50.0. Per-task scores are parsed from the figure, and the column averages reproduce the stated overall figures.

The downstream results are both labelled by humans or oracles:

• Selection: oracle selection over 16 candidates raises complete factual success from 26.7% to 90.0%.
• Refinement: in an 8-input study, human-guided recursive refinement meets all seven requirements for all 8 inputs. Parallel Oracle@8 does so for 3 inputs.

Limits

• Downstream numbers are upper bounds. Both selection and refinement use human judgments, not a trained verifier, so they show headroom only. The fetched text reports no model trained on the 14,126 examples.
• The explanation Judge is text-only. It never sees the 3D evidence and only checks semantic match to the reference explanation. The Judge model is not named in the fetched text, though its validation by human inspection is said to be in Sec. 5.2.
• Some tasks are small. Fine-grained Geometry has 60 examples and Motion has 134. Identity uses only synthetic FaceScape faces.
• Only GPT-6 Astra is named in the fetched text. Other models' results are not shown.
• The fetched text is truncated partway through Section 4. Licence, release format and code are not stated; check the project page at https://ai-yeslab.github.io/3DVerifier-page/.