Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading

What it claims

SemiAnalysis reports that DeepSeek-V4.1-Flash's Engram tables, about 189 GiB in their configuration, run as well from pinned host DRAM as from HBM. On B300 with the week-0 stack, moving the table back into HBM gave results within run-to-run variance. The authors credit async prefetch and overlap work for this. The freed HBM lets a B300 replica drop from TP4 to TP2, which improved the pareto curve by up to 1.6x. SSD offload on B200 lost to DRAM at every measured point. The report also gives Engram ablations showing the component cannot be cleanly removed at inference.

How it works

Engram lookups are addressed by token IDs, not hidden states. The runtime can therefore prefetch rows while earlier layers compute. DeepSeek-V4.1-Flash reads 24 rows at each of two Engram layers. That is about 12.4 KiB per token position across the model, or 3.1 KiB per GPU at four-way split.

flowchart LR
  A[Token IDs] --> B{Table location}
  B -->|HBM| C[GPU kernel reads HBM]
  B -->|Pinned DRAM| D[Same kernel via UVA]
  B -->|SSD mmap| E[CPU: copy IDs, dedupe, gather]
  E --> F[Pinned buffer to GPU]
  C --> G[Dequantize on GPU]
  D --> G
  F --> G

The HBM and DRAM paths share one GPU kernel that selects and dequantizes rows. With Unified Virtual Addressing it reads pinned host memory directly, and both paths support full decode graphs. The SSD path is an unoptimized vLLM fork using memory-mapped files, run without GPUDirect Storage. It round-trips row IDs to the CPU between segments of the GPU execution graph. A warm page cache removes the disk reads but keeps this coordination, which is why a RAM-cached file still loses to a pinned DRAM table.

Evidence

The B200 comparison is measured on total tokens per dollar against P90 interactivity. Near 125 tokens/s/user, DRAM gives 121 million total tokens per dollar and SSD gives 52 million. Every SSD point has a DRAM point with both higher interactivity and more tokens per dollar.

DeepSeek did not release the original paper's trained Engram models, so the authors replicated the setup on fineweb-edu. Each run cost about 6E18 FLOPs. They reproduced the U-shaped scaling and the gains over pure-MoE baselines.

Ablations on DeepSeek-V4.1-Flash:

• Suppressing Engram worsens token likelihood in every evaluated domain, most for encyclopedia text and several code corpora. GSM8K accuracy stays within run-to-run variation.
• On teacher-forced CRUXEval, answer loss rises from 0.2848 to 0.3093 bits/token without Engram. Forcing the original expert routing raises it to 0.3375. Expert rerouting partly compensates for the missing memory.
• Removing Engram during prefill or decode lowers CRUXEval accuracy and lengthens outputs. Keeping it for prefill only preserves more accuracy than keeping it for decode only.

A gate-score scan of high-use n-grams finds names, code fragments, relational phrasing, licences and boilerplate.

InferenceX results cover H100, H200, B200, B300, GB200, GB300 and MI355X. SemiAnalysis puts MI355X at 2-4x worse performance per dollar than B200 on day 7, after TCO normalization. On its first release image, MI355X was up to 42x worse than B200/B300. NVIDIA vLLM ran on all six NVIDIA SKUs on day 0; AMD's documented ROCm image was not public for the first 23 hours.

Limits

Strong gate scores do not identify cache-hot rows. Skipping low-gate reads saves nothing, because the gate needs the retrieved key. Skipping reads would require a separate usefulness predictor before retrieval. The SSD result reflects an unoptimized path without GDS, and the measurement does not break down time per operation. The authors say SSD offload pays only if it enables a cheaper server or extra useful capacity, and this path provides neither. Ablation degradation measures this trained model's dependence on Engram, not the gap against a model trained without it. The "CUDA moat" framing and the claim that 4-hi HBM gives the lowest cost per token are SemiAnalysis's own views. The fetched text ends before the mechanism section on LongCat and Qwen3.8 Flash Next.