Knowledge That Is Linear Before Fine-Tuning Generalizes After It

What it claims

Before fine-tuning, you can measure how linearly readable a relation is in a model's subject representations. That measure ("Pre-linearity") predicts whether a new relation built on top of it will generalize to held-out subjects after fine-tuning. It predicts this better than whether the model can recall the relation. The headline per-model Pearson r ranges from 0.87 to 0.92, with a median of 0.90. The page adds the number the summary omits: when recall, number of answers, number of fine-tuning subjects and subject pretraining frequency are controlled for jointly, r is still 0.70 to 0.82. The setting is eleven open-weight models, base and post-trained, from 2B to 27B parameters, over 70 relations.

How it works

• Answers are scored by first token only. Answers sharing a first token are merged into one "effective answer."
• New relations are built by relabeling.
• Label sets come from a single WordNet 3.0 noun category, such as noun.animal.
• A deterministic algorithm keeps common, non-hyponymous names whose first tokens are distinct under every model's tokenizer.
• It also drops names that are taxonomic neighbours of the relation's subjects or answers.
• A random bijection then maps each original answer to a label.
• The split keeps at least four fine-tuning subjects per answer and at least one held-out subject. Held-out subjects are never used to fit, select or tune anything, including the probe.
• Fine-tuning:
• All weights are updated (full fine-tuning) for 20 epochs.
• Loss is computed on the label's first token only.
• Prompts use a fictional frame ("In the novel XYZ, each entry in the Breviary has an emblem showing one animal…"), so the model learns a fact about a novel rather than a false world fact.
• Four paraphrased training templates are used, plus a fifth template reserved for evaluation.
• Every model reaches accuracy 1 on its fine-tuning subjects.
• Metrics:
• All metrics use "candidate accuracy": top-1 restricted to the relation's answer set, class-balanced by averaging per-answer accuracy, with chance at 1/|E|.
• Recall is the pretrained model's candidate accuracy on the original relation.
• Generalization is the fine-tuned model's candidate accuracy on held-out subjects, under the unseen template.
• Pre-linearity is the gain over chance of a cross-validated cross-entropy linear probe on pretrained subject representations.

Evidence

• Within-model correlation between Pre-linearity and Generalization is r = 0.87 to 0.92 (median 0.90) across 70 relations, for each of the 11 models.
• With all four confounds controlled jointly, r = 0.70 to 0.82 (Table 1).
• The correlation is essentially unchanged when the bijection and fine-tuning seed are re-randomized, and when animal labels are swapped for labels from 13 other WordNet categories.
• Error analysis: a wrong label is mapped back to the original answer it encodes. These recovered answers are more often among the probe's top answers for the subject than among the pretrained model's own top answers (Figure 4). The authors report the probe predicts fine-tuned errors better than the other explanations they test.

How to use it

The practical use is as a cheap pre-fine-tuning diagnostic. Fit a linear probe for the source relation on subject representations, and expect weak held-out generalization where probe gain over chance is low, even if the model recalls the facts well. The paper says all prompt templates are in the supplementary material under artifacts_v1/prompts/. The fetched text names no code release, licence or specific model list.

Limits

• The fetched text cuts off partway through the Pre-linearity definition. The probe's layer and dimension choices, the model names, the per-model results tables and any limitations section are not visible here.
• Everything is measured on first tokens, with answers restricted to a closed candidate set. Free-form generation is not evaluated.
• The tasks are single-hop relabelings of existing relations under a fictional frame with full fine-tuning. Parameter-efficient fine-tuning, multi-hop or compositional facts, and models above 27B are not covered in the visible text.
• The results are correlational across relations. The text claims predictiveness, not that linear structure causes generalization.