LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

What it claims

LeapQuant quantizes the recurrent state of delta-rule linear attention (Gated DeltaNet, KDA) to 8 bits, training-free and without calibration data. Across 12 model–task pairs it matches FP32 accuracy while cutting state memory traffic by 3.4× and end-to-end memory footprint by up to 56%. The memory saving matters most with prefix caching, where a separate state is kept for each cached prefix. The authors identify two error sources:

• Recurrent accumulation: each update starts from an already-rounded state.
• Outliers: large values are concentrated in a few rows and columns and widen the quantization range.

Each technique targets one of these.

How it works

The method covers recurrences of the form S_t = Diag(α_t)S_{t-1} + k_t(v_t − S_{t-1}^T β_t)^T, with β=0 for models without the delta rule. Appendix A maps other linear-attention variants onto this form.

flowchart TD
  A[Quantized residual + compensator tokens] --> B[Decode token: buffer α,k,u in high precision]
  B --> C{End of window?}
  C -- no --> B
  C -- yes --> D[Reconstruct full state S_p]
  D --> E[Fit r rank-one compensator tokens]
  E --> F[Smooth residual key rows]
  F --> G[Quantize residual]
  G --> A

• Per-window quantization. The quantized boundary state stays fixed for a window of p tokens. Each token's (α, k, u) tuple is buffered in higher precision, and each output is computed from the fixed state plus the unrolled buffered updates, which is the same chunkwise form used in prefill. The full state is re-quantized only once per window, so quantization frequency drops by a factor of p. The experiments use 16-token windows.
• Compensator Tokens. At each window boundary, r rank-one terms k̃ũ^T are fitted to the state by Frobenius least squares and kept in high precision. Only the residual is quantized. These terms have the same form as a real-token update with identity decay, so the existing decode kernel processes them ahead of the next window's tokens with no extra kernel. They produce no output, and they are discarded and refit at every boundary.
• Residual smoothing. Each key row is scaled by c = sqrt(mean |row|), with a small floor, before quantization. The scales are stored with the residual and recomputed each window.

Evidence

• State error vs. FP32 (Qwen3.5-9B, PG-19, 64K decoded tokens):
• Per-token quantization error grows steadily with context, even for BF16.
• Stochastic rounding lowers BF16 error by only 3× and makes INT8 diverge.
• Per-window quantization alone lowers error at 64K by 100× for BF16 and 39× for INT8.
• Full LeapQuant at 8 bits stays below per-window BF16.
• Outlier reduction (one head, layer 13 head 16, 8K context): the largest value falls from 176× the median, to 47× after compensator tokens, to 5.3× after smoothing.
• Overhead: with r ≤ 8, the exposed kernel overhead of fitting compensators is within 7% in all settings. Compensator reconstruction is hidden behind the state memory read.
• Why it matters: on GLM-5.3-Flash-NVFP4 on B200 at batch size 256, linear attention takes a roughly constant share of decode time as context grows.

How to use it

The fetched text gives no code, repository link, kernel names, or licence. The kernels build on the chunkwise formulation used by Triton/FLA. Nothing beyond that is stated about integration.

Limits

• Related quantizers don't transfer. Quamba2 and Q-Mamba are specialised to Mamba's selective SSM. Any architecture outside the stated diagonal-decay plus rank-one delta-rule form needs checking against Appendix A.
• The fetched text is truncated partway through residual smoothing. The per-benchmark accuracy tables, the ablations over window size p and rank r, and any results below 8 bits are not visible here.
• Only 8-bit is established. That is the only bit-width the abstract and introduction report as near-lossless.
• Narrow error evidence. The error-growth and outlier figures each come from a single model (Qwen3.5-9B), and the outlier figure from a single head.
• Status: the paper is a v1 arXiv preprint dated 29 Sep 2026.