When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Training

What it claims

Beyond the conceptual frame the summary gives, the page nails down the setup and the headline shift. The verifier is used as a candidate evaluator (it scores learner-generated responses for a given image/question without ground-truth access), not as an answer generator. AAG's whole justification is a precision gain: on MMMU, stricter chosen-confidence selection raises chosen-response precision from 0.52 to 0.87, while symmetric score-spread filtering at matched coverage actually drops it from 0.52 to 0.47 — evidence that the gain is the asymmetric criterion, not just filtering harder. Across all six MMMU configs, AAG shifts mean held-out effect from −1.42 to +0.98 points (paper text) / −.014 to +.010 in the Table 1 accuracy units.

How it works

Standard self-DPO: for each prompt, sample K=4 candidates, verifier picks y⁺ (max score) and y⁻ (min score), optimize DPO. AAG replaces the usual score-gap filter with an absolute-confidence gate on the chosen response only:

• Quantile form: keep pairs where s_V(x,y⁺) ≥ q_{1−α} (empirical (1−α)-quantile of chosen-response confidence).
• Threshold form: keep where s_V(x,y⁺) ≥ τ, default τ=0.9, fixed without held-out labels.

Theorem 2 says that at fixed coverage α, selecting the largest posterior p(z)=Pr(y⁺ correct) maximizes chosen-response precision; AAG approximates this given only rank consistency of verifier confidence (weaker than calibration). The reason gap-filtering fails: a large margin says y⁺ beats y⁻, not that y⁺ is reliable. The Safe-Transfer certificate is arccos F + arccos T < π/2; a non-positive margin is deliberately inconclusive, not a failure prediction. Reference directions g_P, g_Q use ground truth and exist only for post-hoc diagnostics — the training algorithm never sees them.

Evidence

• Models: learners Qwen3-VL-4B, Qwen2.5-VL-7B, Qwen3-VL-8B; frozen verifiers Gemini-3.1-Pro (G3.1), Gemini-2.5-Pro (G2.5) → six configs. β=0.1, lr 1e-5, seeds {17,42,123} for 4B.
• Data: MMMU 900 examples split 700 self-improvement / 200 sealed held-out; MathVista control 500 transductive + 300 held-out. Held-out metric is macro-avg accuracy over 30 subject groups.
• Table 1 held-out means vs frozen: Raw −.014, AAG +.010, Correct-pref (ground-truth diagnostic) +.008 — notably AAG matches or exceeds the ground-truth-ordered reference. Raw hurts held-out on Qwen2.5-VL-7B (both verifiers) and Qwen3-VL-8B/G2.5 (.533→.484).
• MathVista positive control (Qwen3-VL-4B/G3.1): transductive .654→.750 (+9.6); Safe-Transfer Margins +10.7° (Raw, F=0.76) and +20.6° (AAG, F=0.86), both with T=0.78.

Limits

• Silent negative transfer is the core warning: MMMU self-training data still improves +3.52 points on average while held-out drops 1.42 — the target-task signal gives no warning.
• Correct-pref. is explicitly not an upper bound on held-out performance, so "beating ground-truth ordering" is a diagnostic curiosity, not a ceiling claim.
• Learners span two generations, so authors caution against reading differences as a scaling trend.
• Held-out sets are small (200 for MMMU, 300 for MathVista); τ=0.9 is a single fixed operating point, and AAG's guarantee rests on the untested-in-general assumption that verifier confidence preserves posterior ranking.