SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

What it claims

Self-speculative decoding has a draft configuration, such as how many layers to skip or how many visual tokens to keep, and the best choice differs between prompts. SP²ec picks that configuration online, per prompt, while generation runs. It treats it as a bandit problem over an ordered set of drafters and assumes throughput rises to a single peak and then falls. It needs no training. It sits on top of an existing self-speculative method rather than replacing it. The paper is a NeurIPS 2026 publication. The fetched text does not include the paper's own results tables, so the only headline number is the abstract's "up to 11%" throughput gain over the base methods.

How it works

• Arms. Each arm is a drafter built from a nested subset of the target model's components (S₁ ⊂ S₂ ⊂ … ⊂ S_K), with equal cost steps between them. The framework is BanditSpec (Hou et al., 2025). In each decoding round: 1. The bandit picks an arm. 2. One speculative-decoding round runs with that drafter. 3. The bandit observes the number of accepted tokens and the elapsed time.
• Objective. The paper defines wall-time regret: the extra seconds needed to finish this prompt, compared with always using the best fixed configuration. A lemma shows that this fixed-best policy is the optimal policy when per-round cost is deterministic.
• Why there is a single peak. Two assumptions, both checked empirically, produce it:
• Mean accepted tokens show diminishing returns as more components are kept.
• Per-round time is affine in the number of retained components, c(s) = κ·s + b. The cost function is treated as known in advance, profiled on a validation set.

Together these make throughput (accepted tokens divided by cost) single-peaked over the arm order.

• Regret guarantee. The leading regret term depends only on the arms next to the optimum, not on the whole arm set.
• cosSim, the authors' own drafter sequence. After one full-model token, it removes the transformer block whose input and output hidden states have the highest cosine similarity, treating that block as the most redundant. It then runs a speculative step, recomputes the scores, and removes the next block. Repeating this gives K nested drafters and avoids searching all 2ᴺ possible skip patterns.
• Positioning against prior work. SWIFT and similar methods choose which layers to skip under a fixed budget. SP²ec chooses the budget itself for each prompt, so the authors describe it as orthogonal to those methods.

Evidence

The evidence in the fetched text is from cosSim on Qwen2.5-VL-7B-Instruct (Figure 2):

• Time per step falls linearly as the skip ratio rises.
• Acceptance rate shows diminishing returns. Keeping about 55% of layers still gives roughly 80% mean acceptance.
• Throughput in tokens/s is single-peaked across skip ratios between 0 and 0.6.
• The best skip ratio is spread across many prompts, which is the paper's case for adapting per prompt.

Evaluation covers Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B. The page names SWIFT, KnapSpec, SpecVLM and cosSim as base methods. Comparisons are against the base self-speculative method and an adaptive-selection baseline, which the page does not name.

Limits

• Results not in the fetched text. The text stops in Section 3. Per-task numbers, benchmarks and hardware are missing; only "up to 11%" is stated. Read the paper's tables before expecting a gain on your workload.
• Cost model assumes short generations. The affine cost model is derived for short outputs, where context length barely changes per-round cost. Robustness for longer outputs is deferred to Appendix I.4.
• Assumptions you must meet. Mean acceptance must be stationary for each arm, the optimal arm must be unique, and the drafters must form a nested, equal-cost-step sequence. Token-selection methods must also fit this ordering.
• Profiling required. The per-round cost must be profiled for your hardware before use.
• No code shown. The fetched text has no code, repository link or licence.