LLM Inference in a Flash! (UC Berkeley)

What it claims

This is a distinct paper from the cited "LLM-in-a-Flash" [4] — the summary's framing (storing weights in SSD to dodge DRAM limits) describes that older work, not this one. The actual contribution targets Compute-in-Flash (compute-in-memory that runs matmuls inside the flash array), and solves the two things that break LLMs on such hardware: no efficient floating-point, and limited write endurance. Two algorithmic pieces: (1) end-to-end integer-only inference including nonlinear ops, and (2) dictionary-based KV-cache compression via sparse coding. On Llama-3.1-8B and Qwen-2.5-7B it reduces dynamic KV-cache write traffic by 15× with limited accuracy loss, and an analytical system model projects 3.1×/2.7× latency/energy savings at short context and 4.4×/6.8× at long context versus a standard DRAM system (for their "S2" configuration).

How it works

Integer-only pipeline: W8A8 (INT8 weights and activations) for linear layers, plus integer approximations for the operators GPU pipelines usually leave in FP. It uses SmoothQuant-style activation smoothing with group-wise (per-group scale) quantization along the hidden dimension. Because group-wise scales break the cross-hidden-dimension reductions in Softmax and RMSNorm, they requantize the inputs to those ops to token-wise INT16 (one scale per token), then use second-order polynomial approximations (as in I-BERT), with right/left bit-shifts to avoid INT32 overflow. RMSNorm's 1/√var uses a Newton–Raphson integer square root; division is avoided via fixed-point reciprocal. They add an IntSiLU approximation to support SwiGLU (LLaMA-style), which I-BERT did not need.

KV-cache compression: each K/V vector is represented as a sparse linear combination over a large, static read-only dictionary resident in flash — exploiting flash read bandwidth and capacity while writing only lightweight sparse codes/metadata, which is what spares the limited write endurance. Adapted from Lexico but with an added hierarchical sparsity scheme to cut read I/O; Lexico targets GPUs, this targets CIM constraints.

Evidence

The 15× KV-traffic reduction and accuracy figures are on Llama-3.1-8B and Qwen-2.5-7B. The latency/energy numbers come from an analytical system-level model under Compute-in-Flash constraints, not measured hardware — worth noting. For context on why endurance matters, they cite prior work (KVNAND, HiFC) measuring e.g. ~1.98 TiB/day of KV writes under sustained long-context workloads against a 1TB TLC drive's 750 TBW endurance.

Limits

• The fetched text cuts off at Section 3.2.1; no results tables, ablations, or accuracy numbers are visible beyond the abstract's "limited accuracy degradation." The concrete metric (perplexity/task scores) is not shown here.
• Speedup/energy claims are modeled, not benchmarked on silicon — the paper presents an analytical model, and the headline numbers are for one specific "S2" configuration.
• Targets specialized Compute-in-Flash / near-flash hardware; there is no indication these gains transfer to conventional GPU serving.