Skip to content

Back to the catalog

From Patching to Pruning Visual Computation in Vision Language Models
arxiv.orgpaper

From Patching to Pruning Visual Computation in Vision Language Models

Patch-to-Prune (P2P) is a training-free framework that repurposes activation patching from mechanistic interpretability into an inference-time compute bypass for vision-language models, replacing unnecessary visual-token projections in certain decoder layers with fixed neutral proxy vectors while preserving sequence length, token order, and residual pathways. Tested on Qwen2.5-VL and LLaVA family models across seven multimodal benchmarks, P2P at 3 percent tolerance retains about 94 percent of dense accuracy while cutting FLOPs by 55 percent, and reveals that intermediate decoder layers do most of the task-relevant visual integration while early and late layers need little token-specific visual computation.

TL;DR Training-free activation-patching method cuts VLM inference FLOPs by 55 percent while retaining 94 percent of dense accuracy, exposing which decoder layers do visual reasoning.
Open original ↗
Document

What it claims

On Qwen2.5-VL-3B-Instruct, the safe-layer tolerance τ trades accuracy for compute smoothly. Average accuracy is 74.6% at τ=1%, 70.6% at 7% and 67.1% at 10%. Compute falls from 2.1 to 1.3 TFLOPs, against a dense reference of 3.1 TFLOPs. The authors pick τ=3% as the default.

The sensitive band sits at roughly decoder layers 14–21 in that model. The safe set starts at the input and output ends and grows inward as τ is relaxed. Accuracy breaks down once the safe set reaches that band.

The redundancy applies only to visual tokens. Patching text-token computation is damaging at most depths, so P2P never bypasses text-token computation.

How it works

The method has three stages:

  1. Prototypes. Run the frozen model on 100 examples drawn from that benchmark's training set. For every layer ℓ, every linear module (W_Q, W_K, W_V, W_O, W_gate, W_up, W_down) and each stream (image or text), record the mean output activation. The mean is taken per example and then averaged across examples, so long sequences and large images do not dominate.
  2. Safe-layer selection. On a separate set of 100 validation examples, run cumulative sweeps: - Forward sweep: patch layers {0…L} with the means, with L growing from the input side. - Backward sweep: patch layers {L…N−1}, with L shrinking from the output side.

A configuration is admissible if its accuracy is at least (1−τ) times dense accuracy. Each sweep stops after two consecutive inadmissible boundaries, and its frontier is set one layer back. The safe set is the union of the forward prefix and the backward suffix. It is then fixed for test time.

  1. Bypassed inference. In safe layers, visual tokens skip the attention and MLP projection matmuls, and the cached image prototype is written in their place. Text tokens, and all tokens in other layers, are computed normally. All tokens stay in the sequence, with token order, positions, attention mask and residual stream unchanged.
flowchart LR
  T[Token at layer l] --> Q{Visual token AND layer in safe set?}
  Q -- yes --> P[Write cached mean prototype]
  Q -- no --> C[Normal projection matmuls]
  P --> R[Residual stream, sequence unchanged]
  C --> R

Optional P2P-aware fine-tuning. LoRA adapters are inserted into the projections of the safe layers. Training runs under the same patched graph, so patched positions pass no gradient through their adapters. The loss is cross-entropy on answer tokens only, with the prompt masked. Training uses the same 100-sample statistics split that built the prototypes. Fine-tuning is excluded from the main results.

Evidence

  • Layer sensitivity analysis: covers Qwen2.5-VL-3B, LLaVA-Next Vicuna-7B and Vicuna-13B on POPE, WhatsUp and ScienceQA-IMG.
  • Main evaluation:
  • Models: Qwen2.5-VL-3B/7B-Instruct and LLaVA-v1.6-Vicuna-7B/13B-HF.
  • Benchmarks: WhatsUp, POPE, ScienceQA-IMG, GQA, TextVQA, CV-Bench and AI2D.
  • Splits: calibration, validation and test sets are disjoint.
  • Baselines: dense, FastV and PyramidDrop, run in one framework with identical prompts, precision and test examples.
  • The authors present the token-pruning methods as complementary rather than competing. They suggest token pruning could be applied inside the critical layers that P2P keeps.
  • Calibration size: the prototype set size of 100 was chosen from an ablation (Figure 5). The resulting numbers are not in the fetched text.

How to use it

Per task you need:

  • the frozen pretrained model, with no weight or architecture changes;
  • about 100 labelled examples to build the prototypes;
  • about 100 more to run the sweeps;
  • one tolerance τ.

The page gives no code, repository, command or licence.

Limits

  • Truncated page: the fetched text ends before the main results tables. The ~94% retention and 55% FLOP reduction come from the abstract. The per-model and per-benchmark breakdown, the comparison numbers against FastV and PyramidDrop, wall-clock latency figures and the fine-tuning recovery numbers are all not visible here.
  • Task-specific calibration: prototypes and safe sets are built separately for each benchmark. Transfer of one safe set across tasks or input distributions is not shown.
  • Single-model tolerance sweep: the τ sweep and the layer 14–21 band are reported for Qwen2.5-VL-3B only.
  • Prefill compute only: savings depend on visual tokens dominating the sequence, and only visual-token projections are skipped.
  • FLOPs are not latency: reductions are reported mainly in FLOPs. Real speedup depends on whether your inference stack can actually skip per-token matmuls for a subset of positions.

Technologies named here

Techniques named here

  • Patch-to-Prune
  • Activation patching

Related in research