Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding

What it claims

Relaxed (lossy) verification in speculative decoding makes completions longer, and that length cancels most of the gain in accepted tokens per round. The authors test five published training-free rules: CACTUS, mentored decoding, reducible fuzzy speculative decoding (r_fuzzy), and the two speculative cascade rules (spec_casc_opt, spec_casc_tok). The extra tokens are draft tokens the target rates as unlikely, and they end up in the reasoning trace. According to the authors, the added length is waste. On GSM8K it follows an answer already written. On AIME it delays the answer. Forcing the trace to end near the generation budget recovers far more correct answers than it loses.

How it works

Each rule is written as a choice of relaxed distributions π that replace the target p at up to three points: acceptance, the residual after a rejection, and the bonus token. Each π has its own parameter α. Because α means different things per method (divergence budget, threshold, deferral cost, mixture weight), methods are compared at matched mean accepted length ℓ, not matched α. The match is approximate.

The accounting uses verifier rounds R ≈ L/(ℓ+1). With length inflation λ = L_relax/L_strict, a relaxed rule saves decoding time only if (ℓ_relax+1)/(ℓ_strict+1) > λ. Since a round accepts at most N_draft tokens, no relaxation of a given drafter can absorb λ above (N_draft+1)/(ℓ_strict+1). The cascade token rule is the only one that can reject tokens strict verification would accept, even at α = 0.

Evidence

The study covers 360 cells: 12 target-drafter pairs, 6 datasets and 5 rules. The targets are GPT-OSS-20B, Qwen3-8B, Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Llama-8B. The drafters are EAGLE-1, EAGLE-3, P-EAGLE and DSpark heads, plus standalone Qwen3-0.6B, Qwen3-1.7B and Llama-3.2-1B-Instruct. The datasets are GSM8K, AIME24, HumanEval, LiveCodeBench, MT-Bench and LongBench-v2. Each ratio to lossless has a 95% paired bootstrap interval.

• Length. Length grows in 248 of 360 cells, with λ median 1.08 and mean 1.54. It varies more by rule (median 0.97 for spec_casc_tok, 1.39 for CACTUS) and by drafter (0.94 standalone, 1.23 EAGLE heads) than by task. At each rule's gentlest setting, median λ is 1.07, against 1.22 at the loosest.
• Rounds. R/R* has median 0.93 and mean 1.10, and 114 cells need more rounds than lossless. Round ratio tracks length (Spearman 0.78) rather than acceptance gain (−0.12). Equation 3 predicts it within 2% at the median.
• Time. On the primary pairs, median time ratio is 1.01 against 0.96 for rounds. Eleven cells save rounds but not time.
• Accuracy. 198 of 300 graded cells lose two or more points. CACTUS, spec_casc_opt and r_fuzzy lose in 143 of 180 cells, by 0.19, 0.14 and 0.21 on average. Mentored decoding loses in 33 of 60 cells (mean 0.03), and spec_casc_tok in 22 of 60 (mean 0.005). Of the 144 graded cells with λ significantly above 1, 132 lose accuracy (mean 0.21).
• Temperature. At Qwen3-8B's recommended temperature of 0.6, three rules stop inflating on GSM8K and LiveCodeBench. spec_casc_opt still inflates by 1.5x.

How to use it

Each rule is a patch to vLLM's rejection sampler, with tests against closed-form values. Code, patches, prompts and per-run outputs are released at https://anonymous.4open.science/r/lossy-token-eff. Runs used vLLM 0.26.0 on one H100 with N_draft = 6, sampled drafts, temperature 1.0 and top-p 1.0. The practical rule for evaluating any lossy method is to report completion length and verifier rounds alongside accepted length.

Limits

• Measurement. Timing is at batch size 1, with one run per case per arm and seed 0. Extra seeds were used only on the two primary pairs. Cut-off runs make λ a lower bound.
• Rule coverage. Four pairs (standalone drafters and P-EAGLE) run each rule at its loosest α only. For the Qwen3-8B EAGLE-3 head, the Llama EAGLE heads and DSpark, CACTUS and spec_casc_tok modify only the acceptance test.
• Scope. Trained relaxation methods (judge heads, learned ensemble weights) are out of scope.
• Truncated text. The fetched text ends partway through Section 3.5, so the Section 4 analysis appears only as stated claims, without its numbers.