ZipFusion: Fusing Lossless KV Decompression with Attention for Efficient LLM Decoding

What it claims

ZipFusion stores completed KV cache pages in a 12-bit-per-value lossless format and has a FlashInfer-based attention kernel restore the BF16 values in registers. No uncompressed copy ever sits in GPU memory. The motivation is a measurement: when existing lossless codecs (SplitZip, IBP, nvCOMP LZ4) are adapted to SGLang and decompress before attention, decompression takes 32.7% to 73.8% of decode time. ZipFusion avoids that pass. It reports 1.23x to 1.25x decode attention speedup over FlashInfer on an H100, and decode throughput gains at 8K context of 26.3% for Llama 3.1 8B Instruct and 25.8% for Qwen3-8B over uncompressed SGLang.

How it works

The format exploits narrow exponent ranges in BF16 (1 sign, 8 exponent, 7 fraction bits):

1. Each page's keys and values, per layer and KV head, are split into independent blocks. With 16-token pages and head dimension 128, that is four 512-value blocks for K and four for V.
2. The base b is the block's minimum exponent rounded down to an even number. A block is compressed only if every exponent lies in [b, b+15]. Otherwise it stays raw.
3. Each value keeps its low byte unchanged (7 fraction bits plus the lowest exponent bit). It adds a 4-bit code: the sign plus d = floor((e-b)/2). That is 12 bits per value, so an eligible block shrinks from 1024 to 768 bytes, excluding metadata.
4. The exponent is restored as b + 2d + r, where r comes from the retained low byte. This is integer bit manipulation, so no rounding occurs.

flowchart LR
  A[KV block] --> B{All exponents in b..b+15?}
  B -- yes --> C[Low bytes + 4-bit codes, 768 B]
  B -- no --> D[Raw BF16, 1024 B]
  C --> E[Attention kernel]
  D --> E
  E --> F[Restore BF16 in registers]
  F --> G[Tensor Core QK^T / PV]

Keys and values use different layouts, matched to their operand roles in QK^T and PV. Each thread therefore finds the eight elements its Tensor Core instruction needs stored together. Code bytes pair values (x0,x4), (x1,x5), (x2,x6), (x3,x7). A mask on one branch and an unpack on the other each yield four high bytes after one 32-bit add of b/2 replicated per byte. The sum never carries into the sign bit or the next byte.

Per KV head, each 4-block K or V group has 8 bytes of metadata. Four bytes give the first block's offset. The other four mark each block as raw or give its b/2, and blocks are located by summing prior sizes.

The memory manager tracks the physical space left for KV writes as pages compress. It frees chunks in one layer for reassignment to another, and reports the capacity to the SGLang scheduler. The scheduler admits new requests while reserving space for running generation.

Evidence

• In the motivating test (Llama 3.1 8B, BF16, batch 32, 2048 input tokens, one H100, six runs of 256 steps), decode steps take 16.2 ms (SplitZip), 37.5 ms (IBP) and 41.8 ms (LZ4). Non-decompression time stays near 10.9 ms in all three.
• At each method's largest supported batch, ZipFusion gives 2.76x (Llama) and 2.61x (Qwen) the throughput of the best lossless baseline.
• Throughput comparisons use the same GPU memory budget, with each system at its largest supported batch.

Limits

• The ceiling is 25% savings per block. The fetched text does not report what fraction of blocks qualify in practice, or the metadata overhead.
• The implementation covers BF16 KV caches only. It is evaluated on one GPU (H100) with two 8B models.
• Only the most recent tokens' pages stay uncompressed until completed. Prefill and initial compression costs are excluded from the Figure 1 timing.
• The fetched text cuts off mid-method. Ablations, longer-context results and any code release or licence are not visible.