Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

What it claims

Rate-Controlled Refresh (RCR) re-selects which prefix (mostly visual) KV entries stay on the GPU during decoding. The paper's starting point is that one-shot prefill selection cannot work for long descriptions, even with perfect scoring. On DetailCaps, a prefill oracle that scores tokens with the uncompressed answer's own attention reaches only .44 to .52 ROUGE-L fidelity. Exact per-step re-selection reaches .48 to .61. On short-answer VQA the same oracle is lossless: Qwen2-VL TextVQA scores .851 against .848 uncompressed. The authors attribute the long-form gap to staleness. The most-attended prefix position stays the same between adjacent steps only 41% (Qwen2-VL) to 80% (LLaVA) of the time.

How it works

Drift is the attention mass a step would put on evicted prefix tokens, averaged over KV heads per layer. A fixed drift threshold fails because drift magnitude is not comparable across samples. RCR keeps drift as the trigger and adapts the threshold per sample.

• Cache layout: [32 selected prefix | 32-token window | generated], which gives 64 KV entries per KV head. BACON makes the initial selection. The full prefix KV stays aside, in CPU memory in a deployment. A refresh overwrites only the first 32 slots. Keys already carry RoPE, so nothing is re-encoded.
• Rate control: one threshold τ per sample, shared across layers, starting at τ0 = 0.03. After each layer it updates as τ ← τ·exp(0.05·(f − ρ)), with target refresh fraction ρ = 0.4. The update tracks the (1 − ρ)-quantile of drift. Layers that drift more still refresh first.
• Detector: the top 8 right singular vectors of each head's prefix key matrix. Logits for evicted tokens are approximated in this 8-dimensional space; kept tokens use exact logits.
• Candidate re-selection: on a trigger, RCR fetches keys for the detector's top 128 tokens (4c), scores them exactly, keeps the top 32, and then fetches their values. That is 160 vector reads, compared with a full-prefix scan.
• GPU-side decision: candidates are computed in every layer and merged under the trigger mask with torch.where, which avoids a host sync. The authors verify that output is token-for-token identical to the branching version.

flowchart TD
  A[Detector estimates drift] --> B{Drift > τ?}
  B -- yes --> C[Top-128 candidates, exact score, keep 32]
  B -- no --> D[Keep old selection]
  C --> E[Masked merge on GPU]
  D --> E
  E --> F[Update τ toward ρ]
  F --> A

Evidence

The tests use Qwen2-VL-7B, LLaVA-NeXT-Mistral-7B and Qwen3-VL-8B with greedy decoding. Tasks are TextCaps (500 images, up to 64 new tokens) and DetailCaps (300 images, up to 512 new tokens). Fidelity is ROUGE-L against the uncompressed output, with paired bootstrap 95% confidence intervals.

• On DetailCaps, RCR improves over BACON by 11.4 to 21.0 points. It refreshes 40 to 48% of layer-steps.
• RCR beats the fixed threshold by 2.0 to 4.8 points on DetailCaps. The fixed threshold needs a τ tuned per setting, between 0.02 and 0.15.
• Tercile analysis on Qwen2-VL DetailCaps: with τ = 0.03, high-drift samples refresh 35% of the time and nearly match exact re-selection (.479 vs .484). Low-drift samples refresh 1% of the time, although exact re-selection lifts them from .462 to .763. The gap to InfiniGen correlates with mean drift (Spearman −0.41) but not with output length (−0.01).
• At this budget Quest gains at most 6.0 points over BACON, and ArkVale falls below BACON on Qwen2-VL.
• Cost per KV head per step is about Np/16 detector dot products plus 160·r CPU reads, where r ≈ ρ. Quest, ArkVale and InfiniGen read 64 vectors per step. RCR's detector holds Np/16 key equivalents on the GPU, against Np/4 for InfiniGen.
• Components were chosen on a 100-image development split, which shares two images with the test set. They were confirmed under a pre-registered decision rule.

Limits

• On TextCaps, outputs last only 12 to 29 steps, too few for τ to converge, so the realised refresh rate overshoots ρ.
• Exact per-step re-selection is not a strict upper bound. RCR slightly exceeds it on LLaVA DetailCaps.
• Reference-based CIDEr and ROUGE-L do not separate the dynamic methods from each other. Only fidelity to the uncompressed output does.
• Per-layer thresholds were not explored.
• Offloading needs a read conditioned on the trigger mask. That discussion (§7), the full cost figure and the video results are cut off in the fetched text.
• The page names no code release or licence.