LLM Inference in a Flash!

What it claims

The engineering that makes "integer-only" actually hold for a modern decoder-only LLM on Flash CIM, which the summary names but does not open up. The hard part is not INT8 GEMMs — it is keeping Softmax and RMSNorm in integer while using group-wise quantization along the hidden dimension. Those two operators reduce across the hidden dimension, and group-wise scale factors make that reduction invalid without dequantizing to float. The paper's contribution is the set of tricks that resolve this tension without an FP fallback.

How it works

Integer-only pipeline, per the algorithm section:

• Linear layers: W8A8 (INT8 weights and activations), plus SmoothQuant-style activation smoothing to push outlier difficulty into weights. Uses group-wise quantization along the hidden dimension (per-group scales), because per-tensor INT8 gives unacceptable error under LLM long-context dynamic range — the authors note this error is far worse than in BERT, so I-BERT's approach alone is insufficient.
• Nonlinears: second-order polynomial approximations à la I-BERT. Since LLaMA uses SwiGLU not GeLU, they build IntSiLU from the same integer exp(·) approximation.
• The reduction fix: at the inputs to Softmax and RMSNorm they requantize to token-wise INT16 — a single scale per token instead of per-group — restoring a valid cross-hidden reduction. Because the second-order approximation can then overflow INT32, they estimate overflow from the INT16 magnitude, right-shift before the approximation, and left-shift the output for mathematical equivalence.
• RMSNorm computes 1/√var via a Newton–Raphson integer square-root routine. Both Softmax and RMSNorm avoid division using a fixed-point reciprocal (e.g. 2^32/x) implemented as 32-bit multiply-and-shift.

The KV-cache side (dictionary/sparse coding) is closest to Lexico, but Lexico targets GPU inference; this work adapts it to CIM and adds a hierarchical sparsity scheme to cut read I/O.

Evidence

This fetched text is the abstract through Section 3.2.1 only. The headline numbers (15× KV traffic reduction; 3.1×/2.7× and 4.4×/6.8× latency/energy on Llama-3.1-8B and Qwen-2.5-7B) are stated but the supporting tables, accuracy figures, and ablations are past where the text cuts off. The savings are qualified as coming from the S2 configuration from an analytical system-level model — not measured on silicon.

For endurance framing, the related-work section cites concrete external figures: KVNAND estimates a 65B model decoding at 3 tok/s for 5 years produces 143 TB of KV data; HiFC measures 1.98 TiB/day of KV writes and notes a 1TB drive's default TLC endurance is only 750 TBW vs 6,000 TBW in a confined 200 GiB pSLC region. These motivate why write traffic, not just capacity, is the constraint.

Limits

• The latency/energy savings are from an analytical model under assumed CIM constraints, not hardware measurement — treat as projected.
• "Limited accuracy degradation" is asserted in the abstract; the actual accuracy tables are not in the fetched text, so the degradation cannot be judged here.
• Evaluated only on two ~7–8B models; no claim about larger models or non-LLaMA/Qwen architectures.
• The INT16 requantization at nonlinears is a precision escape hatch — it is the load-bearing step, and whether it holds at longer contexts than tested is not shown in this excerpt.