Long Live the Short King: Why 4-hi HBM Wins

What it claims

SemiAnalysis argues that 4-hi HBM gives lower cost per token than 8-hi or 12-hi for bandwidth-bound inference decode, once a system has enough aggregate memory for weights and KVCache. The report says the industry is already moving this way. Nvidia's Rubin Ultra drops to 192GB per GPU from 288GB on Rubin and B300 by moving to 8-hi stacks. Frontier-lab ASIC teams want 4-hi from HBM4 onward. The authors frame 4-hi as maximizing tokens per HBM wafer: the same bandwidth per cube means 2x the harvestable bandwidth per wafer versus 8-hi and 3x versus 12-hi, before counting better packaging yields.

How it works

HBM4 and HBM4E have 2048 data I/Os per cube, split across the core dies, with at most 512 per die. Four dies is therefore the minimum height that uses all 2048 I/Os. Price follows GB per stack. At 13 Gbps, one HBM4E stack gives 3,328 GB/s. With 32Gb dies, a stack holds 16GB (4-hi), 32GB (8-hi) or 48GB (12-hi).

The roofline model sets concurrency to the lower of two limits. One is the number of users' KVCache sets that fit in memory. The other is the number of users that bandwidth can serve at a minimum tok/s/user SLA. Throughput is users times tok/s/user. Capacity is "stranded" when bandwidth per token cannot read all of it. The authors say larger models shift the curves so taller stacks pay off at higher interactivity, because each decode step reads the weights once whatever the batch size.

Evidence

Roofline, Kimi K3 (2.8T, MXFP4, 1,561GB per replica) on Rubin Ultra NVL576, decode pool only. HBM4E at 13 Gbps gives 128/256/384GB per GPU for 4/8/12-hi, with weights at 16.8GB per GPU. Above 105 tok/s/user, 12-hi adds nothing over 8-hi. Above 213 tok/s/user, nothing beats 4-hi. Peak throughput gains over 4-hi are 8% for 8-hi and 10% for 12-hi. Modeled all-in system cost rises 12.1% for 8-hi and 26.3% for 12-hi, so both lose on cost per token at every interactivity point.

Hypothetical model 3x the size of K3, with 3x KVCache. 12-hi delivers 47% more tokens than 4-hi and 8-hi delivers 36% more, both at about half the speed. 8-hi beats its TCO increase below 180 tok/s/user. 12-hi never beats its cost increase over 8-hi.

InferenceX run, Kimi K3 on 16 GB300. The setup was disaggregated, with 8 prefill and 8 decode GPUs. HBM utilization was cut from 92% to 85%, which removes about 20GB per GPU of KV storage. Aggregate KV budget fell from 53GB to 34GB (36%), and disaggregated from 44GB to 24GB (44%). With KVCache offload to DRAM, throughput matched the full-HBM run until concurrency passed about 70. At that point KV occupancy hit 100% and throughput fell by almost 30%. The restricted run made over 10x more DRAM KVCache reads.

Limits

The roofline uses peak bandwidth. The authors say achieved bandwidth is much lower, which moves the breakeven interactivity points left. The experiment cut only 8% of HBM, not the 50% of an 8-hi to 4-hi move. The results depend on SemiAnalysis's HBM price and Nvidia margin assumptions.

The authors note the obvious objection: the analysis applies a current model to a future system, and servers run for over 5 years. Their rebuttals are forecasts, not data. They argue that parameter scaling matters less now and point to looped transformers, which they say GPT-6 Astra uses. They also argue that models will adapt to the installed hardware base. Pre-training is more capacity-sensitive and is not covered by the cost-per-token result. The memory-supplier impact, SKU decisions and supply detail are paywalled or sit in the separate Memory Model.