Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

What it claims

Speculative decoding on Hopper GPUs buys latency, not energy, once batching has already amortised the weight read. Across 5 drafting schemes and 11 models, speculation cuts mean time between tokens at single stream by 1.3 to 4.3×. It is also the only way some models meet tight per-token targets. Without it, gemma-4-31B-it misses a 25 ms target at every batch tested, and 4 of the 11 models miss 10 ms; with speculation, all of them meet the target. On dense models, verification, not drafting, dominates the energy cost. The authors argue that the quantity to track is accepted tokens per verification pass, weighed against every drafted token. The acceptance fraction alone is the wrong measure, because only accepted tokens divide the weight read.

How it works

The roofline argument rests on measured hardware figures:

• Balance point. The H100 balances arithmetic and memory near 316 operations per byte. The H200 balances near 163, from a measured 4.44 TB/s and 722 TFLOP/s.
• Decode intensity. Single-stream decode runs at about 10 operations per byte, so energy follows bytes moved.
• Per-operation cost. On a GH200, an HBM byte costs about 93 pJ against about 0.5 pJ for a 16-bit tensor-core operation.

Speculation raises operations per byte by verifying several tokens per weight read. At large batch, block drafters push verification past the balance point into the compute-bound regime, where the saving turns into a loss.

The schemes are n-gram lookup (run on the host in SGLang), MTP heads, EAGLE3, DFlash and DSpark. Draft heads draft one token per sequential small pass. Block drafters draft a fixed trained block in one pass: 16 tokens for z-lab DFlash, and 7 for DeepSpec DFlash and DSpark. The engine will not verify a shorter block than the trained size. As a result, block drafters verify more tokens per request than heads on the same model.

The 11 models cover four attention families:

• GQA: Qwen3-4B, 8B, 14B and Qwen3-Coder-30B-A3B.
• GQA plus sliding window: gemma-4-12B, 26B-A4B and 31B.
• MLA: GLM-4.7-Flash and JoyAI-LLM-Flash.
• Gated DeltaNet hybrids: Qwen3.6-27B and 35B-A3B.

Five of the 11 are Mixture-of-Experts (MoE) models.

Evidence

• Batching and per-user latency. On dense models, time between tokens grows 9 to 34% from batch 1 to 32. On MoE models it grows 2.5 to 4.0×, because each added request pulls in its own experts' weights.
• CUDA-graph bias. Running with CUDA graphs disabled credits speculation with launch overhead that graph capture would have removed. The inflation is under 10 percentage points on every dense model and 19 to 39 points on every MoE model at single stream.
• Ranking flips. CUDA graphs can change which scheme wins. On Qwen3.6-35B-A3B and gemma-4-26B-A4B-it at single stream, DFlash saves the most energy with capture off, and the draft head saves the most with capture on.
• Acceptance by text type. Every drafter accepts fewer tokens on conversation than on code. Every trained drafter accepts fewer on Romansh news than on English news.
• DSpark scheduling. DSpark's confidence-scheduled verification still verifies the full block at every batch on the H100. Its saving at serving batch is a few points lower than the default mode.

The setup reproduces production serving:

• Software: SGLang 0.5.12 or newer, CUDA 13.2, CUDA graphs on, BF16 weights and KV cache, greedy decoding.
• Runs: at least 3 seeds, 512 generated tokens per request, prompts drawn from 96-prompt pools.
• Hardware: H100 SXM5 (94 GB, 700 W). JoyAI-LLM-Flash and the Qwen3.6 models ran on an H200 (141 GB).
• Baseline: every speculative run is paired with a non-speculative run at the same batch and settings.
• Energy: read from the NVML board counter at 10 Hz and taken over whole generations, never split by phase. The draft's share comes from removing the draft.

The authors wrote their own SGLang DSpark implementation for gemma-4-12B-it. It reproduces the baseline's greedy output token for token at batch 1.

Limits

• GPU board only. Energy covers the GPU board alone, not the host CPU, memory, network or cooling. This favours n-gram drafting, which runs on the host.
• Closed batches. Batches are submitted together with no further arrivals, which is not continuous-batching traffic.
• Thinking text. Qwen3, Qwen3.6 and GLM-4.7-Flash ran with default chat templates, which generate thinking text, so their results describe that text.
• Coverage. Batch sweeps used the code pool. Conversation and long documents got their own sweeps, but Romansh and English news ran only at batches 1 and 16.
• Unequal verify widths. Each drafter verifies its released token count, so comparisons between schemes mix drafter quality with verify width.
• DVFS. The authors report little headroom for latency-preserving DVFS, but only on the one model they tested.
• Truncated page. The fetched text stops partway through Section 3. Per-scheme energy savings (Tables 3 and 4), the roofline counter results and the drafter-choice guidance are not included.