When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

What it claims

When a spatial predicate in a caption is swapped (e.g. "on" → "beneath"), CLIP ViT-B/32's [EOS] text representation barely changes, and a probe cannot recover the new relation from it. At the final layer (L12), spatial EICS is +0.070±0.012. A bag-of-words probe on the same captions scores +0.372±0.022, so CLIP keeps about 19% of the lexical signal. For colour flips, CLIP keeps 94% (+0.312 vs +0.332). The model therefore discards relational information that is plainly present in its own input tokens. Benchmark accuracy will not show you this.

How it works

• CASP-VL construction. The edit is made on Visual Genome scene graphs, not on caption text, so each pair differs in exactly one concept:
• Relation flip: swap the predicate from a fixed list of alternatives. This changes one sentence.
• Colour flip: change the colour of one object instance. This changes every sentence that mentions that instance.
• Captions. Each caption renders the first five relationships of the scene graph, one sentence per relationship, with the template "A [attr] [subj] is [pred] a [attr] [obj]." The image is never modified, so the perturbed caption is a counterfactual description of the same image.
• EICS. An ℓ2-regularized logistic-regression probe is trained per layer on hidden states of the original captions only. EICS is the rate at which it predicts the new label on the perturbed captions, minus 1/|C|. A score of 0 means chance.
• Baseline. The identical probe is run on normalized word-count vectors.

flowchart LR
  G[VG scene graph] --> E[Edit one concept]
  G --> X[Original caption]
  E --> XP[Perturbed caption]
  X --> H[Text encoder layer ℓ]
  X --> B[Word counts]
  H --> P[Probe trained on originals only]
  B --> P2[Same probe: BoW baseline]
  XP --> P
  XP --> P2
  P --> S[EICS: predicts new label?]
  P2 --> S

Evidence

Data and splits

• 2,400 colour triplets (12 classes × 200) and 1,491 spatial triplets (16 classes, up to 100 each).
• Train/test is an 80/20 split grouped by image, repeated over 5 seeds.
• One-directional antonym flips whose new label never appears in training (e.g. near → far from) are excluded as unscorable.

Representation sensitivity

• Mean cosine similarity between original and perturbed L12 [EOS] states is 0.992 for relation flips and 0.943 for colour flips.

Layer profile

• Colour is barely decodable through L10, then jumps in L11–L12.
• Spatial EICS peaks mid-network (+0.097 at L04) and stays far below BoW at every layer.

Robustness

• Across all splits, CLIP keeps 81–102% of the BoW signal for colour and 16–22% for spatial relations.
• Removing perturbed captions that still mention the old colour (seed 42): colour EICS is +0.391, above BoW's +0.326. Spatial is +0.074 vs +0.366 for BoW.
• Using single-sentence captions (only the edited sentence) narrows the spatial gap: CLIP keeps 52% of BoW for spatial and 98% for colour.
• The pattern holds for CLIP ViT-L/14 and ViT-L/14@336, and with human-written VG descriptions or LLM paraphrases in place of the template.

Other models

• SigLIP-B/16 and LLaVA-1.5-7B were also evaluated. Their numbers are in a table cut off in the fetched text.

Predicate tokens

• The page states the relation is decodable at the predicate's own tokens and lost under [EOS] or mean pooling. The supporting numbers are in §4.3, which is not in the fetched text.

How to use it

• Code and data are at https://github.com/Shubhf/CASP-VL. The page gives no licence or install commands.
• The practical implication: if your pipeline relies on CLIP-style pooled text embeddings for retrieval or filtering where spatial relations matter, expect "X on Y" and "X beneath Y" to embed nearly identically.
• The authors suggest using the released code to test whether compositional training methods such as SGVL, CS-CLIP or MACCO raise EICS. They have not tested this themselves.

Limits

• Text side only. Only the text encoder is probed, and images are never perturbed. Visual grounding is covered in an appendix that is not in the fetched text.
• Template captions. The main results use template captions built from the first five relationships. Repeated VG annotations can leave the old label elsewhere in the caption, and the authors analyse this in an appendix.
• Linear probes only. EICS measures linear decodability from a probe that never sees perturbed captions.
• Truncated page. The fetched text stops partway into §4.2. Cross-architecture numbers, predicate-token results, the comparison against compositional benchmarks, and all appendices are not visible here.
• Status. This is a v1 preprint posted 9 Oct 2026.