STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

What it claims

Uniform quantization of Delta-rule recurrent states fails because each update starts from the previous quantized state, so its error compounds. The paper reports that mean accuracy drops sharply at 4 and 6 bits on both models, with a gap remaining even at INT8. The motivating figure: in official SGLang deployment, Qwen3.8-27B's FP32 state pool exceeds its BF16 weight memory at 70 concurrent requests, even with short contexts. STEPQuant compresses Qwen's recurrent states 6.93× at 4-bit and 5.03× at 6-bit (5.08× for Kimi at 6-bit). At 6-bit, total serving memory including weights falls 68.7% on Qwen and 53.7% on Kimi. The custom kernels give up to 2.91× faster state updates.

How it works

Two components. The bit choices are fixed offline; the scales are computed per step.

Lifetime-aware Bit Allocation (temporal).

• A "unit" is a whole head in Qwen (GDN, scalar gate) and a key row in Kimi (KDA, channel-wise gate).
• Calibration estimates two things per unit:
• distortion d_u(b) at each candidate bit width
• mean log retention ℓ_u of its gate
• The lifetime weight is L_u = Σ_{j<H} exp(2jℓ_u).
• Bits are chosen to minimize Σ L_u·d_u(b_u) subject to the average-bit budget.
• The highest-risk units are kept in FP16 as sparse "pivots".
• Pivots and bit widths are set during offline calibration and fixed across requests.

Key-Row-Aware Dual-axis Fitting (spatial).

• Each entry is stored as X̂_ij = r_i·c_j·z_ij.
• The row-impact score is ω_i = E[g²_{t,i}], where g_t = A_tᵀ q_t. It measures how much error in key row i reaches the readout.
• The row scale is r_i = m_i^{1/2} w_i^{-1/2}, where m_i is the row's mean |value| and w_i is the normalized impact. Large rows get a wider range; high-impact rows get finer resolution.
• Column scales c_j are fitted by minimizing impact-weighted reconstruction error.

flowchart LR
  subgraph Offline calibration
    A[Gate retention ℓ_u] --> C[Bit widths + FP16 pivots]
    B[Distortion d_u b] --> C
    D[Row impact ω_i]
  end
  subgraph Each decode step
    E[Float state update] --> F[Row scales r_i]
    F --> G[Fit column scales c_j]
    G --> H[Quantize to assigned bits]
  end
  C --> H
  D --> F

Evidence

• Lifetime predicts error. Across 2304 Qwen heads under uniform INT6 on C4, gate half-life correlates with accumulated state error (Spearman ρ ≈ 0.80).
• Row impact predicts damage. Key rows were ranked by ω_i into eight groups, and one group at a time was quantized to INT4. Higher-impact groups generally caused larger perplexity increases.
• Outliers sit on both axes. The max-to-median RMS contrast is 10.3× along key rows and 19.4× along value columns. Both exceed 3× in 98.6% of sampled states, and the outliers persist across 2048 decoding steps. This is the case for scaling on two axes rather than one.
• Accuracy. Tested on seven long-generation and six short-generation tasks.
• 6-bit closely matches FP32-state accuracy on both models.
• 4-bit beats uniform INT8 on both models. It nearly matches FP32 on Qwen but leaves a modest gap on Kimi.
• Per-task tables are not in the fetched text.
• The paper also evaluates combining state quantization with weight quantization. Those results are not in the fetched portion.

How to use it

Code: https://github.com/Dreamer-Toby/STEPQuant. The method is integrated into SGLang with custom GPU kernels.

Adopting it requires an offline calibration pass per model to fix the bit allocation, pivots and row-impact scores. The fetched text does not give the calibration set size or the install commands.

Limits

• Only tested on two hybrid gated-Delta-rule models: Qwen3.8-27B (GDN) and Kimi-Linear-48B-A3B-Instruct (KDA).
• "6-bit" is a nominal average budget. The FP16 pivots and stored scales are part of the actual footprint, which is why compression is 5.03× rather than about 5.3×.
• The 4-bit setting is weaker on Kimi than on Qwen.
• The error-propagation bound assumes ‖k_t‖ ≤ 1, β_t ∈ [0,1] and gates between 0 and I.
• The fetched page is truncated mid-method. Implementation details, ablations, the full result tables and the paper's own limitations section were not available.