DLite: History-free Speculative Decoding for Agentic AI

What it claims

DLite is a draft model for speculative decoding that keeps no draft-side KV cache. It drafts each block only from the target model's hidden states and the current anchor token. The authors (Shanghai Jiao Tong University, Shanghai Innovation Institute, Tsinghua, Infinigence-AI) target agentic workloads: long inputs, short outputs, many turns and a growing context. They report two problems with existing drafters there:

• Acceptance length collapses. EAGLE, DFlash and DSpark drop to 1 to 2 accepted tokens on SWE-Bench and AgentLongBench, from much higher values on GSM8K and MBPP.
• Cache upkeep is costly. Maintaining the draft KV cache costs 10% to 20% of end-to-end latency. They measured this on DeepSeek-V4's multi-token prediction module in SGLang.

DLite is trained for Qwen3-4B and Qwen3-8B. It reports acceptance lengths of 2 to 6 and up to 70% over DSpark.

How it works

Each speculative round is rebuilt from scratch:

1. Inputs. Hidden states from one or more target layers each get a learnable depth embedding, which marks them as features rather than tokens and identifies the source layer. These are combined with the anchor token and K mask placeholders.
2. Backbone. A parallel backbone uses bidirectional attention with no KV cache. It uses local position IDs, not global ones, so the problem size stays the same as the sequence grows. It predicts all K positions in one pass.
3. Sequential head. A low-rank, RNN-style head folds the earlier draft tokens of the current block into a recurrent state. It restores only the dependencies inside the block, since earlier context already arrives through the target hidden states. DSpark projects every position to full-vocabulary logits and then refines them. DLite works in hidden space and applies the D×V vocabulary projection once, at the end. The authors say this matters under high-concurrency serving.
4. Verification. The target model verifies the drafts in the standard way.

flowchart LR
  T[Target hidden states + depth emb] --> B[Parallel backbone, no KV cache]
  A[Anchor + K placeholders, local positions] --> B
  B --> S[Low-rank sequential head]
  S -->|next draft token| S
  S --> V[Target verification]
  V -->|new anchor + hidden states| T

Training has two stages:

• Stage one distils from an "isomorphic history-aware teacher", which bridges the structural mismatch between DLite and the target.
• Stage two aligns DLite directly with the target model.

A figure compares direct target distillation using KL and CE losses. The authors say direct distribution-level distillation is non-trivial for this architecture.

Evidence

The fetched text has no results tables. Its numbers are the abstract claims (2 to 6 acceptance length, up to 70% over DSpark) and the motivation figures above. It reports no wall-clock speedup, batch size or hardware. The paper gives a speedup model, τ·Time_T(1) / (Time_T(N) + Time_D(N)), so acceptance length alone does not set the end-to-end gain.

How to use it

Code is at https://github.com/SJTU-DAI-Sys/DLite. Weights are in the Hugging Face collection https://huggingface.co/collections/SJTU-DAI-Sys/dlite. The page gives no install commands, licence or serving-engine integration details.

Limits

The fetched text stops partway through the training section (4.2), so per-benchmark results, ablations and stated limitations are not visible. Only Qwen3-4B and Qwen3-8B targets are named. The design needs access to the target model's intermediate hidden states at every round.