FlatCake: A Flat KVCache Storage Architecture for LLM Serving

What it claims

FlatCake replaces the DRAM-over-SSD KV cache hierarchy used by Mooncake, LMCache and SGLang HiCache with a flat layout. In it, host DRAM and NVMe SSD sit at the same level. The argument rests on profiling eight production traces from Mooncake, Qwen Bailian and Claude Code. These traces break both premises of tiering.

• Temporal filtering fails. In seven conversational and API traces, 53.2 to 94.0% of blocks are accessed only once. Reuse distances have heavy tails: Qwen to-B API has a mean of 2,550 intervening requests and a P99 of 54,728. One-shot blocks flush reusable prefixes out of DRAM.
• Footprints exceed DRAM. The Claude Code trace reuses 83.4% of blocks at a mean distance of 0.85 requests. Each request spans about 3,421 blocks (218,922 tokens). On DeepSeek-V4-Flash in FP8 that is a median of 4.1 GiB across eight ranks, reaching 16.0 GiB at P90, which overruns host DRAM across concurrent sessions.

In both regimes, most hits come from SSD. The authors conclude that migrating blocks between DRAM and SSD adds data movement without keeping hot data in DRAM.

How it works

The design relies on KV blocks being write-once, immutable and restored together per session. Three components implement it inside SGLang:

• FC-GVA (Global Virtual Address Manager). It binds each block to DRAM or SSD once, at creation, within a session-contiguous scope. The block's medium, node and byte offset are encoded in an immutable 64-bit address. Lookup is O(1) with no tier-by-tier search. Nothing migrates, nothing is written back, and SSD space is reclaimed per session rather than in FIFO buckets.
• FC-MAS (Medium-Aware Request Scheduler). It keeps separate queues for DRAM-ready requests and SSD-preparing requests, which prevents head-of-line blocking. Background SSD prefetches are ordered by a size-aware response ratio. Prefetched requests are re-admitted by original arrival time to avoid starvation.
• FC-CIO (Coalescing Direct I/O Engine). Blocks are written in layer-major order. Reads are coalesced, collapsing O(layers × blocks) scattered reads into O(layers) sequential GPUDirect Storage transfers. These stream into a bounded GPU HBM window, overlapped with layer-wise compute, with no host staging buffers.

flowchart LR
  R[Prefix hit] --> G[GVA lookup O1]
  G -->|DRAM medium| D[Fast queue]
  G -->|SSD medium| S[SSD prefetch queue]
  S --> C[Coalesced GDS reads]
  C --> W[Bounded HBM window]
  D --> GPU[Prefill]
  W --> GPU

The baseline's failure mode explains the design. Tiered restores hold host staging buffers for each in-flight prefix, so demand scales as O(context × concurrency). Under load the staging pool runs out, prefetches abort, and blocks already on SSD get recomputed.

Evidence

Claude Code trace, DeepSeek-V4-Flash (284B hybrid MoE), NVIDIA B200

• FlatCake's hit rate is 94.2 to 94.6% across arrival rates. The trace's prefix-reuse ceiling is 94.7%.
• Mooncake's hit rate falls from 87.9% at low load to 64.3% under staging contention.
• At peak load, FlatCake cuts mean TTFT by 1.6x and median TTFT by 1.8x, and raises total token throughput by 1.6x.
• It reaches 1.9x higher SSD read bandwidth and 5.8x lower prefetch latency.
• It removes Mooncake's 6.5x to 6.7x SSD read amplification.

Llama-3.1-70B (dense GQA), NVIDIA A100, 8K to 128K contexts

Mean TTFT drops by:

• 2.3x to 4.9x against prefix recomputation
• 1.5x to 2.4x against Mooncake
• up to 1.8x against Mooncake-GDS, which already bypasses host DRAM

The Mooncake-GDS result points to layout, not just skipping DRAM, as the source of the gain.

Limits

• Truncated text. The fetched text ends partway through §3.2. The rule for choosing DRAM or SSD at creation, the full evaluation and any limitations section are not visible.
• No adoption details. The visible text gives no code release, licence or configuration details.
• Narrow baselines. The production-trace result uses only one model and one trace (Claude Code). Compared systems are Mooncake, Mooncake-GDS and recomputation. LMCache and SGLang HiCache are discussed but do not appear in the visible results.
• Hardware dependency. The read path depends on GPUDirect Storage support.