BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization

What it claims

BiLoCo replaces SVDQuant's BF16 low-rank correction with ±1 sign vectors plus one BF16 scale per rank-one term, on top of an NVFP4 residual. The paper is by Jin, IslamBouli, Goel, Guo and Kim (MIT), posted October 2026 and marked as NeurIPS 2026. A binary rank-r correction costs r(m+n+16) bits, against 16k(m+n) for BF16 rank k. Binary rank 512 therefore fits the vector budget of BF16 rank 32. Batch-size limits on the kernel matter more for deployment than the batch-1 figure in the summary.

How it works

Factors come from greedy "peeling" rather than SVD:

1. Start with R = W. For each component, draw random signs v, then alternate u = sign(Rv), v = sign(Rᵀu) for T = 5 steps.
2. Set α = uᵀRv / (mn), subtract αuvᵀ from R, and move to the next component.
3. After all r terms, refit every α jointly by solving an r×r ridge-regularized least-squares system. This is O(r³) and takes milliseconds per layer.
4. Quantize W minus the correction to NVFP4 (about 4.5 bits per weight) once.

The sign alternation and joint refit come from Riasanovsky and Kazdadi. The authors claim only the NVFP4 application and the fused kernel as new. The theory shows that each sign update is monotone and that the refit coefficients are optimal for the chosen signs. It does not bound output error or perplexity.

The fused kernel builds on a CUTLASS SM100 block-scaled FP4 GEMM with 128×64 output tiles. The batch is zero-padded to 128 rows, and the kernel runs one CTA per SM. GEMM CTAs run on tensor cores. The remaining CTAs compute the sign-bit additions on CUDA cores and write the scaled results to a reused, epoch-tagged buffer, so no clearing launch is needed. The two results are summed in the GEMM epilogue.

flowchart LR
  X[BF16 input x] --> Q[FP4 quantize x]
  Q --> TC[FP4 GEMM on tensor cores]
  X --> CC[Sign add/subtract on CUDA cores]
  CC --> S[Scale by α, epoch-tagged buffer]
  TC --> E[GEMM epilogue sum]
  S --> E
  E --> Y[BF16 output]

Evidence

The paper uses γ, the fraction of a weight matrix's Frobenius norm captured by the low-rank part, measured on 189 projection matrices at equal vector storage:

• Qwen3-4B: 52.9% for binary rank 512, against 29.3% for SVD rank 32.
• Qwen3-14B: 47.3% against 26.3%.
• Llama-3-8B: 50.6% against 27.9%.

The binary advantage is anti-correlated with the spectral ratio σ₁/σ₃₂ (Pearson −0.65 to −0.81).

The perplexity claim is that BiLoCo matches or improves BF16 rank-32 SVDQuant on WikiText-2 and C4 across five models. The fetched text gives no perplexity numbers.

Latency on B300 is measured against the fastest FP4-only kernel the authors could build at each shape:

• At batch 1, BiLoCo adds 6–11% for ranks 16 to 512.
• FP4 GEMM time stays flat up to batch 128, but the CUDA-core branch grows with batch size.
• The largest batch within 20% overhead is 8 at rank 128, 4 at rank 256 and 2 at rank 512.

How to use it

Code is at https://github.com/jindavid/BiLoCo. The page gives no install or run commands. The kernel targets Blackwell (SM100, B300) with NVFP4.

Limits

• Small batches only. Overhead rises quickly with batch size, so BiLoCo suits small-batch decode, not large-batch serving.
• Layer width cap. All CTAs must be resident at once, so N/64 must be smaller than the SM count. That is 148 on B300, which caps output width at under 9,472.
• γ is a proxy. The authors note that γ only stands in for model quality.
• Incomplete text. The fetched text cuts off mid-kernel section. Per-model perplexity tables, accuracy benchmarks and ablations were not available.