Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

What it claims

Copying pretrained Softmax ViT attention weights (Q, K, V projections) into a linear-attention ViT gives no reliable gain over random initialization. For TTT it is consistently worse. A pretraining-free analytical init, Mimetic, systematically beats it. What does transfer:

• MLP weights, by direct copy.
• Attention behaviour, by distilling attention outputs. Matching attention maps, as in prior Attention Transfer work (Li et al., 2024), is the wrong target.

The authors report that copied MLPs plus output-distilled attention let linear ViTs match or exceed the Softmax ViT. They frame this as the opposite of the Softmax-to-Softmax finding that attention alone suffices.

How it works

flowchart LR
  S[Pretrained Softmax ViT] --> A[Attention]
  S --> M[MLP blocks]
  A -->|copy weights| X[No reliable gain]
  A -->|distill outputs| L[Linear ViT]
  M -->|copy weights| L

• Sources: DeiT checkpoints of matching size (Tiny, Small, Base).
• Targets: five linear variants: ELU and ReLU kernel attention (Katharopoulos et al.), InLine, MHLA and TTT.
• ELU, ReLU and InLine have weight shapes identical to Softmax attention.
• MHLA and TTT add components. Their unmatched parameters use each method's default init.
• Distillation loss: matches attention outputs, which is the same choice LoLCATs and RADLADS make in language models. The visible text does not specify the loss form or its weighting.

Evidence

Training setup: the Xu et al. (2024) recipe, 300 or 600 epochs depending on dataset scale, averaged over 3 seeds.

Datasets:

• Tiny and Small: CIFAR-10/100, STL-10, Food, Flowers and Pets.
• All three sizes: ImageNet-1K.

Attention-only copy, ViT-Base, ImageNet-1K top-1 (Table 1):

DINO, MoCo v3 and iBOT sources land in the same band. No pretrained source gives a reliable advantage.

Out-of-distribution, ViT-Base, Random → Copy (Table 2):

Copying attention adds no robustness.

Practical takeaway: if you are porting a Softmax ViT to linear attention, do not spend effort on copying attention projections. Copy the MLPs and distill attention outputs instead.

Limits

• Headline result not shown: the fetched text ends partway through Section 4. The numbers behind the main claim (copied MLP plus distilled attention matching or surpassing Softmax) and the MLP-only ablation do not appear. Check the full paper for the margins.
• Not a cheap conversion: every experiment uses a full 300 or 600 epoch training run, not a short adaptation phase.
• Narrow source: the main source is DeiT only. Model sizes stop at Base, and Base is evaluated only on ImageNet-1K.
• No release information: the visible text mentions no code or weights.