ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

What it claims

On looped Transformers, KV states for the same token are highly similar across loops. ResidualQuant stores each non-final loop as a 2-bit residual against the final loop's KV. On Ouro-1.4B, MATH500 with group size g=32, it reaches 76.0% against 75.0% for BF16, while KV storage drops to 19.3% of BF16. Plain INT2 at nearly the same storage (15.6%) scores 27.8%. The paper also reports that the residual principle carries over to activation quantization, where it beats LoopQ and FlatQuant. Only the section heading for that result is in the fetched text.

How it works

For loop r, the cache stores Q(X_r − α_r·R_r) and reconstructs X̂_r = α_r·R_r + Q(residual) on demand. Quantization is group-wise affine, not symmetric, so INT2 gets all four levels instead of three. The paper compared three choices of reference R_r:

flowchart TD
  A[Choose reference per loop] --> B[Previous-loop: chain]
  A --> C[First-loop anchor]
  A --> D[Last-loop anchor: chosen]
  B --> E[Loop 4 needs 4 cache reads]
  C --> F[Anchor + own residual: 2 reads]
  D --> F
  D --> G[Current token held in BF16 until final loop]

• Previous-loop references are slow. They chain reads: in a 4-loop model, reconstructing loop 4 needs the anchor plus three residuals. Shared-anchor reconstruction (First- or Last-loop) gives up to 1.62× decode throughput over chained reconstruction.
• Last-loop beats First-loop on error. With uniform INT2, average key NMSE (×10³) is 69.80 for Last-loop and 82.57 for First-loop. Last-loop error is much lower in loop 1 (53.55 vs 179.39). The cost is that the anchor loop itself is the worst at 178.20.
• LS scaling. α_r = ⟨X_r, R_r⟩ / ‖R_r‖² is computed per layer, token and head. It accounts for KV norms that vary across loops.
• Rotation. Residuals are rotated with OptR-H: head-wise orthogonal rotations, Hadamard-initialized and optimized to minimize attention output error.
• Mixed precision. Precision is set per loop, mixing INT4 and INT2. The fetched text cuts off before saying which loops get INT4.

Evidence

Ablation on Ouro-1.4B, MATH500, g=32. Each step adds one component:

• Accuracy gap. Against rotation-based KV quantization at the same memory budget, the paper reports up to 13.0% higher accuracy. This covers Ouro-1.4B and Huginn-3.5B on math reasoning and code generation benchmarks; the full table is not in the fetched text.
• Throughput on an RTX 5090.
• At 16k context with a fixed batch, decode is 2.73× faster.
• Batch capacity doubles, which yields 4.15× peak decode throughput.
• Figure 1, at 8k context, instead reports 3.27× peak throughput with 4× batch capacity.
• The baseline is BF16 FlashAttention.

How to use it

The page gives a project page: https://seal-kaist.github.io/projects/ResidualQuant. It does not mention a code release, licence or install steps. Two practical requirements are stated:

• OptR-H rotations must be optimized per head.
• The serving path must keep the current token's KV in BF16 and defer quantization until the final loop finishes (described in their Section 4.2).

Limits

• The fetched text is truncated partway through Section 3.2. The mixed-precision assignment, the per-model and per-benchmark tables, and the activation-quantization results are not visible here.
• Only two looped models are evaluated (Ouro-1.4B, Huginn-3.5B), and throughput is measured on one GPU (RTX 5090).
• The 80.7% figure is theoretical KV storage.
• Throughput claims differ by setting: 3.27× with 4× batch at 8k (Figure 1) versus 4.15× with 2× batch at 16k (text). Check which applies to your context length.
• The method depends on inter-loop KV similarity. The paper notes that Huginn-3.5B can show input-dependent trajectories that form orbits rather than settling to fixed points.