Skip to content

Back to the catalog

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
arxiv.orgpaper

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

ASPIRE is a non-synchronized batched self-speculative decoding framework for long-context LLM inference, where requests independently choose when to verify instead of sharing a global draft-verify schedule. It combines a unified mixed forward pass, a per-request online speculation scheduler using acceptance-rate estimates and a batch-aware cost model, and an intra-draft refresh layer that periodically applies full attention to reduce staleness. Across three models and five reasoning/long-context benchmarks it reports 1.70-4.58x decoding throughput speedup over autoregressive baselines, ~27% better than prior self-speculative baselines.

TL;DR Asynchronous per-request speculative decoding scheduling gives up to 4.58x faster long-context LLM decoding than autoregressive baselines.
Open original ↗
Document

What it claims

ASPIRE's headline gain over autoregressive decoding is not uniform — it scales with context length. The summary gives the 1.70–4.58× range; the page shows the shape: on Qwen3-1.7B it climbs from 2.06× (AIME) to 4.57× (LongBench-v2, 80–100k), and the two named prior baselines it beats by ~27% average are MagicDec (window attention for drafting) and Vegas (attention-score-based context selection, but refreshes only after verification). The paper is accepted at COLM 2026; code is at github.com/Amir-zsh/ASPIRE.

How it works

The scheduler the summary names runs a concrete decision per request per step: keep drafting while draft length d_i < γ_i, else verify. The target length γ_i maximizes a closed form built from the per-request acceptance-rate estimate α_i and a batch-aware cost ratio c_i — roughly argmax (1−α_i^{d+1})/(1−α_i) / (1 + c_i·γ).

The intra-draft refresh design rests on a diagnostic finding the summary omits: they compared attention-map "predictor" sources by mass recall@100 against an oracle. Temporal reuse (same layer, earlier token) decays fast as token lag grows; cross-layer reuse (same token, earlier layer) is far more stable. But per-layer offsets would need many refresh signals, so ASPIRE uses a single late-layer full-attention pass from the previous token — one token old, one full-attention layer per draft step — which matches deeper same-token refresh at lower cost.

Evidence

  • Per-model speedups (Figure 1): Qwen3-1.7B 2.06–4.57×; Qwen3-8B 1.70–2.74×; DS-LLaMA-8B 1.80–2.65×. Larger models show smaller gains.
  • Draft-length heterogeneity study on 256 LongBench samples, 16–20k prefixes: per-request average accepted draft length ranges 2.56 to 20.00, mean 11.51 (drafting 20 tokens, up to 1k generated). This, plus within-request fluctuation across verification rounds, is the motivation for per-request scheduling.
  • Refresh-layer recall improves from layer 0 to 2 then plateaus — additional full-attention layers give little benefit.

Limits

Page text is the paper body up to §4.1; the ablations, exact per-benchmark tables, and end-to-end serving overhead (scheduler cost, latency vs. throughput) are not in the fetched portion. Speedups shrink markedly on the two 8B models, so the 4.58× figure reflects the smallest model on the longest context, not a typical case. All results are throughput speedups over autoregressive/self-speculative baselines; no accuracy-parity data is shown in the text here beyond the claim that verification is exact.

Techniques named here

Related in inference