MPQ: A Message-Passing View of Post-Training Quantization

What it claims

MPQ is a drop-in replacement for the discrete solver stage of weight-only PTQ. It is not a preprocessing method. The authors frame GPTQ, LDLQ and block coordinate descent (QuantEase, CDQuant) as zero-temperature limits of a Gibbs posterior over codebooks. Each of those solvers commits one block to one codeword before its neighbours are seen. MPQ runs at finite temperature first, so near-tied codewords survive until the end. The paper reports the largest gains at 3-bit scalar and 2-bit lattice precision. It also says MPQ composes with activation-aware scaling (AWQ-style) and incoherence processing (random Hadamard rotations). The paper is accepted at NeurIPS 2026, and code is at https://github.com/poseidonchan/MPQ.

How it works

The solver works on one output row at a time. The row is split into blocks of dimension d, and each block has a finite codebook. That covers scalar grids (d = 1) and E8P12 from QuIP# (d = 8, 2^16 codewords).

1. Take a compact SVD of the calibration activations X, so every first-stage step uses an r×n rank-space matrix instead of X.
2. Run VAMP for T_vamp iterations. Each iteration alternates two steps: - A blockwise discrete denoiser computes a softmax over codewords, anchored to the original weights by a Gaussian prior of width τσ_b. - A Gaussian linear module solves a ridge-like system using the Hessian.
3. Adapt the inverse temperature β by empirical Bayes inside the loop.
4. Project to hard codewords by blockwise marginal MAP. The authors note this need not be the joint MAP.
5. Run exact block coordinate descent on the original objective for up to S_CD sweeps, stopping early when no block changes.

flowchart TD
  A[Rank-space SVD of X] --> B[Discrete denoiser: soft codeword probs]
  B --> C[Gaussian linear module + adapt beta]
  C -->|t < T_vamp| B
  C -->|done| D[Marginal MAP projection]
  D --> E[Exact block CD sweep]
  E -->|any block changed| E
  E -->|no change| F[Quantized row]

The authors state that the VAMP stage does not replace local search. Its role is to start coordinate descent from a better basin.

Evidence

The paper gives a union bound showing that the per-block MAP error decays exponentially in the fixed-point precision γ★₁. This holds in an idealized right-orthogonally-invariant surrogate, and the paper also gives an exact formula for how diagonal scaling changes the linear-module error. The paper reports a diagnostic consistent with the bound. The dense E8P12 codebook gives a lower terminal precision than scalar codebooks, and coordinate descent updates a much larger fraction of its blocks. The fetched text contains no perplexity or accuracy tables, so the size of the gains over the baselines cannot be checked from this page.

How to use it

The required inputs are calibration activations, per-block codebooks and scales, and the hyperparameters τ, T_vamp and S_CD. The paper gives no install commands or default values in the fetched text. Appendix B.5 contains Python-style pseudocode.

Limits

The theory assumes an orthogonally invariant operator. Finite calibration Hessians are not exactly that, even after Hadamard rotation, so the authors treat the analysis as qualitative guidance. The analysis also uses the τ → ∞ limit, while MPQ itself runs at finite τ. For dense codebooks the bound is uninformative and the remaining work falls to coordinate descent. The method is evaluated only on dense Llama, Qwen and Mistral models. The fetched text reports no solver runtime or cost relative to GPTQ.