DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

What it claims

Decomposed VQA metrics (DSG, Soft-TIFA, Gecko) score every generated question against a fixed "yes" answer. When the right answer is "no", as with a negation like "no chair in sight", they penalise a faithful image, and their accuracy on negated prompts drops below chance. DEPICT drops the fixed reference. It scores each question by how likely the image-based and caption-only answers are to agree, weights each question by how decisively the caption settles it, and averages the result with a VQAScore-style holistic score. The metric is training-free: one VLM does question generation and all scoring.

How it works

flowchart LR
  C[Caption] --> G[VLM: yes/no questions]
  G --> V[Answer on image: p_v]
  G --> T[Answer on caption only: p_t]
  V --> A[Agree: p_v·p_t + (1-p_v)(1-p_t)]
  T --> A
  T --> W[Weight: abs p_t − p̄_t]
  A --> D[S_dec weighted mean]
  W --> D
  C --> H[Holistic p yes given image, caption]
  D --> M[0.5·S_dec + 0.5·S_hol]
  H --> M

1. Generate questions. The VLM writes yes/no questions directly from the full caption. There is no DSG tuple parsing, dependency graph or post-hoc filtering, and no fixed question budget, so the number of questions N varies per caption.
2. Answer twice. Each question is answered once with the image (p_v) and once from the caption alone with the image removed (p_t). Probabilities are full-vocabulary softmax mass summed over token variants (yes, Yes, yes). They are not renormalised over yes/no, so off-target outputs lower the score.
3. Score agreement. The rule is a_k = p_v·p_t + p̄_v·p̄_t, which rewards matching "no" answers. It reduces to Soft-TIFA's rule ("Faith", a = p_v) when p_t = 1. The paper also names an intermediate rule, "Confirm" (p_v·p_t), which fails on "no" answers.
4. Weight by commitment. Each question gets weight w_k = |p_t − p̄_t|. Questions the caption cannot decide fall to near-zero weight, which acts as a soft quality filter.
5. Merge. S = (1−λ)·S_dec + λ·S_hol, with λ = 0.5 and ε = 10⁻³. Both values were fixed a priori and not tuned on any benchmark.

Evidence

Results are human correlation, SRCC/PLCC. Baseline numbers marked † in the paper are cited from other publications; the rest were re-run.

• Against the fine-tuned evaluator on the same backbone (Qwen3-VL-4B): DEPICT beats DynEval on GenAI-Bench and RichHF. It trails clearly on TIFA160 (0.691 vs 0.802), and still trails there even on the 27B backbone.
• Against VQAScore on the same backbone: the SRCC gain is small, 0.630 vs 0.628 on GenAI-Bench and a tie on RichHF. Most of the gain shows in PLCC and on TIFA160.
• Negation: the agreement rule raises negation accuracy from 19% to 88%. The benchmark is NegBench COCO-MCQ.
• Statistics: score differences come with paired BCa 95% intervals over 10,000 bootstrap replicates resampled by item.
• Backbones tested: eleven in total.
• Qwen3.5 at 4B, 9B and 27B
• InternVL3.5 at 4B, 8B and 14B
• Qwen3.5, 3.6 and 3.8 at 27B
• Gemma-4 E4B, 12B and 31B

How to use it

Prompts are in Appendix A.2. Per caption, the cost is one question-generation pass, two answer passes per question (image and caption-only), and one holistic pass. The scoring passes need token-level probabilities, so an API that exposes only text output won't work. Pass counts and latency are in Appendix A.3, which is not in the fetched text.

Limits

• No code yet. The paper says only "Code will be available for research purposes." Note the usage restriction this implies.
• TIFA160 is a weak spot. DEPICT does not beat the strongest fine-tuned metric there.
• Small margin over VQAScore on the same backbone. The gap on SRCC is slight, so it is worth checking whether the extra passes pay off for your use case.
• Truncated text. The fetched page stops at Section 5.1, so these were not available:
• the Winoground results
• the ablations (question generation, commitment weighting)
• the λ sensitivity analysis
• the per-backbone trends