Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

What it claims

Adaptive DHiSparse adds draft-model speculative decoding (DSpark, DFlash2) to HiSparse-style hierarchical sparse KV serving. It also replaces HiSparse's fixed per-request GPU cache with residency that grows and shrinks with load. Beyond the summary's headline numbers:

• At 100K context and C128, GLM-5.3 + DFlash2 is 1.27× non-speculative Adaptive HiSparse.
• Mean TTFT is 2.1× lower than HiSparse.
• Under HiSparse's latency bounds, output is 2.3–3.7× higher, depending on the TPOT bound.
• The speculative state costs 0.060 GiB of extra required GPU memory per GLM session while supporting the full configured verification block.

How it works

Memory split. Each request and KV source gets a fixed hot cache of B rows, a mutable tail and metadata, plus verification staging when a drafter is present. These are never lent. Complete pages of imported history can additionally sit in reclaimable "optional" capacity, subject to required + optional + admission reserve ≤ pool capacity.

Resident-first admission. A new request keeps its imported history on device and defers host backup while HBM is spare.

Reclamation and refill. When an admission leaves a deficit, the runtime walks requests in reclaim order and takes whole requests' optional pages until the deficit is covered. Other requests keep their pages. If optional pages can't cover the deficit, the admission waits. Pages are freed only after host backup completes and earlier readers finish. Refill uses bounded round-robin. The invariant is that every row is either ready on the GPU or has a host copy.

Resolution. The resolver checks steps in order, stopping as soon as all selected rows are found:

1. The tail and the ready optional-history map.
2. A scan of the hot cache, bounded by B (with LRU replacement).
3. A copy from pinned host memory into the hot cache.

Because the scan is bounded by B, a large optional history does not enlarge the LRU scan. The copy kernel scales from 4 to 32 thread blocks with batch size. On GLM-5.3 at C8 without speculation, selected rows resolved 69.7% in step 1, 30.3% in step 2 and 0.05% from host.

Speculative state.

• Verification queries share missing-row retrieval rather than each fetching its own.
• The drafter keeps only its visible window, in a ring buffer.
• Target layers reuse verification staging once their consumers complete.

Model-specific reuse. DeepSeek-V4.1-Flash shares one physical global KV source across compatible layers, so Reuse layers inherit the prior location vector. GLM-5.3 shares sparse indices but keeps distinct per-layer payloads, so those positions are prefetched rather than shared as bytes.

Evidence

• GLM-5.3, 32K input, 8K output cap (HiSparse's own eval lengths):
• Adaptive DHiSparse gives 1.7–1.9× HiSparse output from C8 to C64, and 1.45× at C128 with 2.3× lower mean TTFT.
• Fully resident + DFlash2 is 5–9% ahead up to C32, while all histories still fit, then stops scaling at C64.
• Load-step on/off experiment: optional residency cuts host-fetched selections from 14.2% to 1.4% at low load. It releases memory under pressure and repopulates as requests drain.
• DeepSeek-V4.1-Flash, 500K context, C512: 1.85× the output of resident serving with DSpark.
• The authors stress relative comparisons under a shared setup. Absolute rates are in their Table 2, which is not in the fetched text.

Limits

• If all histories fit in HBM at your concurrency, plain resident serving with a drafter is slightly faster (5–9% at ≤C32 on GLM-5.3). The gains appear once memory caps admission.
• Rows generated during decoding always follow hot-cache rules. Keeping the full imported history on device therefore does not make every lookup direct.
• Admissions can stall if reclaimable optional pages cannot cover the deficit.
• Evaluated only on GLM-5.3 and DeepSeek-V4.1-Flash, with DSpark and DFlash2. The design relies on each model's layer-reuse structure.
• The fetched text is truncated mid-Section 3. It does not mention code, a release, a licence or the hardware used. Check the full paper for these, for Table 2, and for the hot-cache budget sweep (Section 4.3.4).