The Alignment Illusion in Multimodal Large Language Models

What it claims

Layer-wise visual-text similarity in MLLMs is routinely read as evidence that the language model integrates visual content — but this paper shows the standard scalar measures (σ₁ leading principal-angle cosine, CKA, SVCCA, MIR) cannot tell a genuine visual stream from noise. Replacing projector-output visual tokens with norm-matched Gaussian noise drops task accuracy by 38–50 percentage points across all 13 models, yet the four scalar scores barely move (medians of the separation score are negative for σ₁, CKA, and SVCCA; MIR near zero). The cause is traced to shared LLM weights, and the fix offered is the principal-angle gap (PA gap) = difference between the top two principal-angle cosines.

How it works

Because visual and text tokens pass through the same Transformer blocks, high similarity has two sources: genuine content interaction, or geometry imposed by shared weights ("weight-induced alignment"). The mechanism is traced in four steps:

1. σ₁ between fixed projector output and text stays near the random reference — near-unity values only appear after the shared pathway.
2. Bypassing the MLP changes σ₁ 3.5× more (median) than bypassing attention; the effect depends on trained weights, not architecture.
3. The MLP down-projection Wₒᵤₜ has dominant output directions: ratio s₁/s₂ sits between 1.2 and 1.6, above a matched random reference (~1.0) in every model.
4. The visual-side principal-angle basis projected onto Wₒᵤₜ's top-10 singular subspace captures 2.8×–13.6× more energy than random.

Proposition 1 formalizes this: a shared W pulling both streams toward a common direction inflates σ₁ with no cross-modal coupling term. Since this component is one-dimensional, subtracting σ₂ from σ₁ isolates multi-directional visual structure.

Evidence

Models: 13 MLLMs from five families — LLaVA-OV, LLaVA-OV-1.5, Qwen2-VL, Qwen2.5-VL, InternVL3 — spanning 0.5B–72B parameters. Accuracy measured on MMBench. Two interventions: (a) Gaussian-noise tokens, (b) structured-but-irrelevant image. The PA gap tracks accuracy under graded corruption more consistently than the scalar scores, and under irrelevant input exposes regimes where internal geometry and task accuracy diverge. PCA retains top-k=30 components (Appendix Fig. 12 tests sensitivity).

Limits

This is a diagnostic-validity paper, not a training method — it offers no way to improve alignment, only to measure it more honestly. Fetched text cuts off at Section 3.3 (the full PA-gap definition and its structured-irrelevant-image results are truncated). Accuracy is evaluated only on MMBench. The core takeaway is methodological: a high internal alignment score is not evidence an MLLM uses the image; such claims need paired task-accuracy evidence under controlled visual interventions.

Note: the arXiv ID (2609.30210, dated Sep 2026) is future-dated relative to typical numbering — treat provenance with mild caution.