DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair

What it claims

DRelay inserts a learned repair step between parallel drafting and target verification. After a parallel drafter such as DFlash proposes a whole block, DRelay revisits each position's top-16 candidates. Using evidence from the entire block, it decides whether to keep the drafted token or swap in another candidate. The goal is to fix the early errors that truncate the accepted prefix.

On Qwen3-4B, its average end-to-end speedup over autoregressive decoding is 3.57×. The comparable figures are 3.11× for DFlash, 3.26× for Domino and 3.30× for DSpark. Across Qwen3-4B, Qwen3-8B and the temperatures tested, it beats the best of those three baselines by 7.8–9.9%. Concurrent-serving throughput reaches up to 4.38× autoregressive at matched concurrency.

Why repair has headroom

The authors measured this before training anything, on greedy traces from DFlash and Domino with Qwen3-4B as the target:

• The right token is usually already in the pool. At the first error, the original top-16 pool contains the target token in 94.0% (DFlash) and 95.1% (Domino) of cases. At Domino's actual served first rejection, it does so in 88.8% of cases.
• Later draft tokens help pick the right candidate. A frozen target model scored each candidate by the log-prob of the next four draft tokens. This picked the correct candidate in 46/86 (DFlash) and 31/69 (Domino) cases. That is 53.5 and 44.4 points better than a control that used a suffix from another prompt. The sample sizes are small, and the setup is an oracle: the four following tokens are known to be correct.
• The oracle ceiling is meaningful. Replacing only the first rejected token with the target token, when it is in the pool, raises mean accepted length by 1.65 tokens (DFlash) and 1.22 tokens (Domino), averaged over all complete blocks.

How it works

flowchart TD
  A[Verified anchor + context features] --> D[Parallel drafter: top-16 per slot]
  D --> G[Global Read]
  G --> S[Causal selector at position i]
  H[Verified target history] --> S
  P[Selected prefix P_i] --> S
  S -->|KEEP| N[Append draft token]
  S -->|REPAIR| R[Append alternative candidate]
  N --> P
  R --> P
  S -->|block done| V[Target verifies single path]
  V -->|accepted prefix + next anchor| A

1. Global Read. Two bidirectional attention–FFN blocks mix the slot features with probability-weighted candidate summaries. Then 4-head candidate-specific attention runs over every (position, candidate) pair. It is biased by the draft log-prob and a relative-position term, so competing candidates can attend to different continuations. A zero-valued null key lets a read be content-free, which means uncertain context does not force an edit. A learned per-slot scalar λᵢ controls how much of the global context is fused in. Global Read uses only the drafter's hidden states and distributions, never unverified future tokens.
2. Causal selector. It runs sequentially across the block. A Gate and a Rank head combine the global context, the verified history and an addressable read of the selected prefix, then choose KEEP or REPAIR. A repaired token conditions all later decisions. The selector can repair any position, not only the first error.
3. Training. The draft backbone and the selector are trained jointly. A candidate-support loss teaches the backbone to put correct tokens in the pool. A repair loss is weighted by each position's potential contribution to the consecutive accepted prefix. Neither the fusion nor the gate is supervised with oracle future correctness.

Evidence

All numbers below are Qwen3-4B end-to-end speedups versus autoregressive decoding on an H800, in the order DFlash / Domino / DSpark / DRelay:

Under SGLang serving, DRelay's average speedup gain is 14.7–16.8% over DFlash, 8.7–9.3% over Domino and 8.1–9.3% over DSpark.

Limits

• Narrow evaluation. Only Qwen3-4B and Qwen3-8B were tested, on a single H800.
• Smaller gains on dialogue. Gains are much smaller on dialogue and instruction-following tasks: about 2.2–2.4× on MT-Bench and Alpaca, against about 3.6–4.5× on math and code.
• Heuristic motivation. The motivating analyses are oracle or controlled experiments with small samples (n = 86 and 69). They show headroom; they do not measure the trained DRelay.
• Retraining required. DRelay trains the draft backbone jointly with the selector, so it is not a drop-in add-on to an existing frozen drafter.
• Missing details. The fetched text cuts off partway through Section 4. Training data, training cost, selector overhead and ablations are not in it. The text also mentions no code, weights or licence.