Execution-Path Qualification and Realized Costs in Speculative Decoding on Consumer Systems

What it claims

Speculative decoding of Qwen3-8B (target) with Qwen3-0.6B (draft), both Q8_0, was slower than plain target-only decoding on almost every tested consumer path. The one exception was a code-only workload on Apple M4. Output was greedy, processed one request at a time, and capped at 128 generated IDs.

The paper's second finding is about correctness. Changing the target's micro-batch size can change greedy output. On Intel Arc B570 (Vulkan), the speculative serialized path did not reproduce the UB128 reference, matching only 60/72 runs. A draft-free target-only UB1 predictor, however, predicted every live token: 72/72 runs, including all 12 runs on 4 reference-divergent prompts. The paper says that predicting the output is not the same as being a lossless replacement for the reference, and that this result does not identify a unique numerical cause.

How it works

• Three distinct agreement endpoints, which the paper says must not be interchanged:
• H: text reconstructed from parsed CLI output (historical).
• Q: captured generated IDs equal a frozen target-only greedy reference.
• P: a sealed prediction equals reserved live IDs.

P does not imply Q.

• B570 prediction protocol: 1. In discovery (12 prompts × 3 runs), UB1 matched 36/36. 2. The hypothesis was frozen. 3. All 72 UB1 ID arrays were sealed. 4. Reserved live runs were then executed.

In an example divergence (HumanEval/134), the first difference is at generated-ID index 88: reference ID 914 versus ID 1946 in all 3 runs.

• Complete-cost accounting: speedup is S = C / (D_real + R_target + X).
• C counts all committed tokens, including correction/bonus tokens.
• D_real is realized draft time, including interface replay.
• R_target is complete target work per round.
• X is residual sync, transfer and bookkeeping.
• All terms are normalized to target-only per-token cost.

Break-even requires C > D_real + R_target + X. The authors warn that acceptance rate, isolated draft throughput and target-forward batching microbenchmarks (V_micro) each measure only part of this loop.

Evidence

All speedups are baseline/speculative, given as the prompt geometric mean with a 95% bootstrap CI. Timing excludes model load and prefill.

Further results:

• RTX qualification: batched matched the reference in 75/90 runs and serialized in 90/90, so only the serialized path qualifies.
• 358H despite draft cost removed: with draft and residual costs removed, 358H still stays below parity at fixed measured progress and target work.
• 358H isolated draft numbers: both 358H paths were slower despite favorable isolated draft measurements.
• M4 energy: package-energy efficiency on M4 HumanEval was 1.013× [0.990, 1.034], so no energy gain is established.
• M4 Guard: Guard beats target-only but loses to fixed speculation.

How to use it

The paper gives no code or commands in the fetched text. Practical takeaways stated on the page:

• Before claiming a speculative path is lossless, check generated-ID equality against your own deployed target-only greedy reference. Batched verification can silently change outputs.
• Measure the complete loop, including correction tokens, interface replay and synchronization, rather than acceptance rate or draft throughput.
• Expect the result to depend on the workload. The same M4 stack went from 0.842× to 1.387× when moving from Spec-Bench to code tasks.

Limits

• Separate studies: each study is frozen with its own comparator, runtime and quantization. The authors say the results do not rank hardware and do not isolate workload-only causes.
• Historical timings use text-path agreement only: these are the M4, earlier RTX and B570 numbers, which are not strict-ID results. The B570 number describes a partially mismatching path.
• ID equality is not correctness: capped ID equality is not functional HumanEval correctness or pass@k.
• Unclear cause and transfer: the B570 predictor shows sufficiency, not cause. RTX's target-only predictor failed discovery (24/36).
• Narrow bootstrap intervals: CIs exclude session, thermal and recalibration variance.
• Provenance: Windows provenance is weaker, and the exact historical identity of the 358H setup is unproven.
• No 358H energy result: no end-to-end energy result is claimed for 358H.
• Deviations: a preregistered synthetic ceiling was not executed.
• Truncated page: the fetched text ends partway through Section 5, so the appendices and later sections were not reviewed.