The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models

What it claims

Researchers at the University of Amsterdam's ILLC (Shihui Li, Mette Andersen, Martha Lewis) locate a "metaphoricity" direction or subspace in the residual stream of seven instruction-tuned LLMs. They report that intervening on it flips literal/metaphoric predictions and shifts free-form generation between literal and metaphorical readings. The models are Gemma 2 9B, Llama 3.1 8B and 70B, Mistral 7B, Mixtral 8x7B, and Qwen 2.5 7B and 72B. The authors call this the first attempt to control metaphoricity through activation intervention rather than prompting or an external model. They also claim the directions are "somewhat consistent" across datasets.

How it works

There are three datasets, all with one metaphorical target word per short, fixed-template sentence:

• NOUN: 130 tuples rewritten by ChatGPT from Fig-QA into "The noun1 is (a/an) noun2", then manually checked.
• ADJ: the 100-pair Tsvetkov et al. (2014) test set, templated as "The N is ADJ."
• DOM: Gutiérrez et al. (2016) adjective-noun pairs across 8 domains, such as temperature and texture. Evaluation uses n=100 held-out samples per steering direction.

A verb dataset was dropped because model accuracy on it was too low to probe.

The tasks are four-shot identification (literal vs metaphoric), four-shot interpretation (apt vs inapt paraphrase, on NOUN) and five-shot generation (explain the adjective's meaning, on DOM).

The method has three stages:

1. Locate. Full-vector activation patching between paired prompts (for example "The skin is dry" vs "The wit is dry") finds the influential token positions and layers. The full stop at the end of the target sentence is strongly implicated.
2. Find the subspace. DAS trains an orthonormal R ∈ R^(d×k) at that token and the layer with the best interchange intervention accuracy (IIA). Training runs for 1,000 Adam steps with model weights frozen and alternates literal→metaphor and metaphor→literal patches.
3. Steer. The DAS subspace is compared against Marks and Tegmark's mass-mean steering (h + α(μ_met − μ_lit)), using means from correctly predicted samples. Classification uses α=1. Generation applies the intervention at every decoded token, following Li et al. (2023), with α ∈ {1, 2, 5, 10}.

Generation is judged by Qwen 2.5 72B Instruct on the first sentence only. A completion counts as degenerate, and scores 0, if any word repeats three or more times. The flip rate counts only samples the judge labelled correctly before steering.

Evidence

The fetched text stops early in the results section. Patching on both parts of speech and both tasks shows three high-influence groups:

• the differing target tokens;
• the final ":" token;
• a third group the text truncates before describing.

The page states that all models perform the baseline tasks "to a reasonable standard" except one Gemma model on DOM, which is excluded. The baseline table, IIA scores, steering accuracies and generation flip rates are not in the fetched text.

Limits

• Results not visible. No quantitative steering or generation results appear in the fetched text, so the size of the causal effect cannot be checked from this page.
• Model naming conflict. The excluded model is called "Gemma 27B" in the results, but the evaluated list names only Gemma 2 9B.
• Narrow datasets. The datasets are small (100–130 items) and syntactically uniform by design. They cover nouns and adjectives only.
• Hedged claims. The authors themselves describe the causal effect as "some" and the cross-dataset consistency as "somewhat".
• Judge-dependent generation scores. Generation results depend on a single LLM judge that sees only the first sentence of each completion.