How GLM5.3 Sparse Attention Affects HBM Memory Usage

What it claims

SemiAnalysis argues that DeepSeek Sparse Attention (DSA) in Z.ai's GLM-5.x models cuts per-step attention reads but not KV capacity, because the top-k selector needs the full context in HBM. The article then uses the September 28 InferenceX AgentX snapshot to compare GLM-5.3 serving cost and latency across NVIDIA and AMD. Which system is cheapest depends heavily on the time-to-first-token (TTFT) limit applied. GB200 wins on cost at 150 tok/s only by accepting p90 TTFT of about 14 to 19 seconds.

How it works

HiSparse (SGLang). HiSparse treats HBM as an LRU cache over host DRAM. Tokens chosen by top-k that are missing from HBM are loaded from DRAM. Cold tokens are evicted back to DRAM. Loading for layer N overlaps execution of layer N-1, a technique taken from HiCache. The cost is miss I/O.

flowchart LR
  A[Indexer selects top-k] --> B{In HBM?}
  B -- hit --> C[Sparse MLA]
  B -- miss --> D[Load from host DRAM]
  D --> C
  E[LRU eviction] --> F[(Host DRAM)]
  F --> D

Model. GLM-5 has 744B total and 40B active parameters. Each token uses 1 shared expert and 8 of 256 routed experts. DSA has two parts:

• Lightning indexer. It scores tokens with a down-projected ReLU dot product, summed with weights across heads. Selection is shared across attention heads.
• Sparse MLA in MQA mode. Top-K is 2048, and attention below 2K tokens stays dense. vLLM switches to MHA mode for about 2K to 5K tokens under data parallel, and about 2K to 77K under tensor parallel.

GLM-5 uses 64 query heads where DeepSeek V3.2 uses 128. That gives a decode arithmetic intensity of 120.8 FLOP/B. SemiAnalysis speculates this targets the Moore Threads MTT S4000 (about 128 FLOP/B). The QK NoPE dimension rises from 128 to 192, which is still a 33% SDPA FLOP cut after the head reduction.

IndexShare (GLM-5.2). Every 4 DSA layers share one indexer. Top-K indices are cached so the shared layers can reuse them. The indexer is trained against the average attention distribution across the shared layers. Reported effect: 75% less indexer cache, 75% fewer indexer FLOPs, and 1.5x to 1.8x throughput, depending on context length and phase.

Post-training. The pipeline runs:

1. SFT.
2. Reasoning RL, then agentic RL, then general RL.
3. On-policy cross-stage distillation from the reasoning and general RL checkpoints.

GLM-5.2 reportedly replaced the distillation step with "parallel OPD", merging more than ten expert models in about two days. The synchronous RL stage uses GRPO without a KL term, plus IcePop masking and Clip-Higher. The asynchronous stage is REINFORCE-like with Direct Double-Sided Importance Sampling. Its IS ratio is computed against the inference policy.

SAO, for long-horizon agentic RL, replaces group advantages with single-rollout GAE from a separately trained value model, skipping tool-output tokens. The value model is updated twice per policy step, has its full-attention layers frozen, and is initialised from a scaled pre-training corpus. SAO doubles the memory footprint. The slime framework's multi-task orchestrator handles 1,000 concurrent rollouts.

Evidence

All cost figures are per million total tokens, including cached input history, under InferenceX's hyperscaler TCO model.

• At 150 tok/s. GB200 (Dynamo-SGLang) costs $0.044 versus $0.049 for MI355X with ATOM, about 12% lower. These values are interpolated from curves, not tested directly.
• Across speed targets. ATOM is 13% cheaper at 100 tok/s. GB200 is 5% cheaper at 125 tok/s and 12% cheaper at 150 tok/s.
• Output tokens only, at 150 tok/s. GB200 costs $5.92 and MI355X ATOM costs $6.68 per million output tokens.
• GB300 versus GB200 (Dynamo-SGLang). GB300 serves 13,950 tok/s per GPU against GB200's 11,873, a 17.5% lead. At assumed prices of $2.31 versus $1.86 per GPU-hour, that lead does not offset the higher cost.
• Tested configs only, ≥150 tok/s and p90 TTFT ≤2 s. MI355X ATOM costs $0.0607 versus $0.0666 for B200 Dynamo-SGLang, about 9% lower. With the TTFT limit relaxed to 10 s, GB300 with Dynamo-TRT-LLM qualifies at $0.0451, 26% below ATOM.
• Host-memory cache reuse (B200). Going from 8 to 16 concurrent requests, GPU cache reuse falls from 90.3% to 54.8% and host reuse rises from 6.0% to 40.3%. The total hit rate stays above 95%.
• vLLM, GLM-5.2, NVFP4. Keeping the first decode step on a consistent CUDA graph path cut average TPOT from about 40 ms to 22 ms.
• ATOM on 8 MI355X, long context. Chunked prefill across pipeline stages gave 98% higher throughput and cut median TTFT from 28.6 s to 8.7 s.
• TileRT. TileRT compiles decode into one persistent kernel, with vLLM handling prefill. FP8 TileRT on MI355X reaches 2x the P90 interactivity of the best FP4 config (the page writes "MI335X"). It reaches 40% higher interactivity than GB300 NVL72.

Limits

• Costs are modelled. They come from InferenceX TCO estimates with assumed GPU-hour prices, and the 150 tok/s comparison is interpolated.
• The text is inconsistent on GB300. It says GB300 has the lowest cost at 150 tok/s, then shows GB300 losing to GB200 on cost under Dynamo-SGLang.
• TileRT is incomplete. Its TTFT is described as suboptimal. KV transfer optimisation, FP4 support and larger batch sizes are still pending.
• Some claims are SemiAnalysis inference. These include the Moore Threads hardware target and the reading of the distillation stage's purpose.
• Some details are undisclosed or unverified. Z.ai did not disclose the size of SAO's value-model pre-training corpus. Z.ai claims storage-backed teacher swapping, but slime PR#1538 implements only CPU-pinned weight swapping (TensorBackuper).
• Part of the article is paywalled. The GLM-5.3 cybersecurity analysis is not included.