Computation and Data Movement for Inference

What it claims

The SemiAnalysis essay argues that MoE serving should be planned around the movement of three things: expert weights, private KV cache and compact activations. Parameter count matters less. It makes several sizing and design claims. A pipeline stage should have about 400 to 500 GB of local fast memory as a 2027 benchmark. Decode batches may be best near one to five requests rather than large. Admission control coupled directly to decode retirement can oscillate. A single aggregated worker per turn is viable only if the chip is strong in compute, memory bandwidth and network at once.

How it works

The four regimes differ in where reuse comes from:

• Prefill is compute-bound. Initial inputs rarely reach 100,000 tokens.
• Midfill appends roughly 50 to 5,000 new tokens to a cached prefix. That prefix can be tens of GB and is commonly beyond 100,000 tokens. Midfill's arithmetic intensity can be hundreds of times that of one-token decode. Its KV traffic can be tens of times that of a same-sized prefill.
• Decode attention gains nothing from batching, because each query scans its own KV cache.
• Decode experts can share weight loads across requests, but only when routed tokens coincide. With about 15,000 layer-experts and 8 to 16 routed experts per token, small batches share little.

In prefill and midfill, route records are sorted by expert ID so that each expert's weights load once per bucket. For example, 128 tokens × 8 experts gives 1,024 assignments over a 256-expert bank. Per-expert arithmetic intensity is approximated as 2nW / (B_W + n·B_A + B_O). Gains flatten once activation and output traffic dominate.

NVL72 matters, the essay says, because 72 synchronized GPUs split a 256-expert layer into about 3 or 4 experts each. This allows fast cycling through experts at the cost of heavy all-to-all traffic. The essay also covers an SRAM-resident "DAF" (disaggregated attention-FFN) variant. It claims up to 100x lower energy per byte than HBM and removes the need to batch, but the network may cost more than the experts it connects.

Pipeline depth stops paying off at the KV crossover. Weights shrink as W/P, but each stage holds KV for P live streams. The essay's stage estimate is: 70 kB/token (described as "conservative" for proven compressed attention) × 2M live tokens ≈ 140 GB. Double buffering raises that to about 280 GB. Weights, buffers and margin bring the total to 400 to 500 GB. Full-attention models need hundreds of kB/token, MLA-like designs tens of kB, and new hybrids about 25 kB. Routed experts hold over 95% of weight memory in a large MoE.

KV state is stored as immutable, append-only blobs. A branch is created rather than rewriting the shared path.

flowchart LR
  H[HBM: active batch only] --> C[CPU DRAM staging]
  C --> P[Shared DRAM pool]
  P --> K{Classifier}
  K -->|likely reuse| S[SSD tier]
  K -->|unlikely| D[Discard]
  D -.->|rebuild from text| H
  S -->|RDMA, striped per GPU| H

The essay states that rebuilt KV may differ subtly from the original, so services vary in how freely they discard.

The essay plots SemiAnalysis "AgentX" conversation traces. They show compaction as contexts near practical limits, which it gives as 250k tokens for Opus 4.8 and 1M for Fable.

How to use it

Section 21 gives a mapping procedure:

1. Measure the four regimes separately, split by context length and workload class.
2. Describe the repeating layer group.
3. Measure active state, not just total parameters.
4. Pipeline the layer stack, use tensor parallelism only for broad indivisible operations such as long-context attention, and spread experts widely with expert parallelism.
5. Map attention first.
6. Place experts according to the expected routing distribution.
7. Keep all-to-all routing and attention reductions inside scale-up. Send pipeline activations and KV blobs over scale-out.
8. Stop at the KV crossover.
9. Sort routes in the fill stages, and keep decode batches small.

Against scheduler oscillation, the essay lists these measures:

• ready-work buffers between stages
• admission control driven by smoothed backlog and predicted service rate
• hysteresis on worker reconfiguration
• capacity reserved for short interactive work
• batch-size changes no faster than their effect can be observed

Limits

The text is cut off at section 22 ("The next view is time"), about 60,000 of 68,759 characters. A worked example the essay says compares midfill and decode parallelism settings is not in the fetched text.

The essay contains no measured benchmarks. All figures are back-of-envelope estimates or opinions:

• the 400 to 500 GB stage memory target
• the 25 to 70 kB/token KV figures
• the judgement that Vera Rubin 72 is strong at midfill but underused in decode
• the 100x SRAM energy figure

The "stalactites" in the Anthropic traces are attributed to unexplained proprietary behaviour. The essay does not compare orchestrators such as Dynamo, Mooncake or vLLM.