Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

What it claims

Gandalf (arXiv 2610.30791v1, 22 October 2026, University of Chicago, TensorMesh and Samsung Semiconductor) argues that SSD KV-cache offloading wears out enterprise SSDs in months, not the five-year warranty period. The authors estimate this from two one-day SemiAnalysis Claude Code traces. Qwen3.5-27B generates 81 to 98 TB of KV cache per day and gets an SSD lifetime of 0.4 years. Kimi-K3 generates 199 to 240 TB per day and gets 0.2 years. The peak concurrent working set reaches 8.6 TB, beyond practical host DRAM. A 30 TB enterprise SSD exhausts its rated TBW in under a year, and a 2 TB drive does so in days.

How it works

The method rests on reuse heterogeneity in hybrid-attention models during multi-round and agentic workloads. Full-attention KV for the preceding context is needed at every later round. Linear-attention state and sliding-window cache are needed only for the next round, so each is reused once. Writing those one-shot states to SSD spends endurance for almost no hits.

The base policy keeps linear-attention state in host DRAM and full-attention KV on SSD. The authors flag two problems with that fixed split:

1. Hardware-to-utilization imbalance. Many concurrent sessions can overflow DRAM with linear state, and long contexts can overflow the SSD with full-attention KV. Gandalf therefore gates SSD admission on the utilization and pressure of each tier.
2. Moving hits from DRAM to SSD can raise TTFT, because SSD bandwidth is lower.

An offline oracle that jointly optimizes admission and eviction gives the design rule. An SSD admission is worthwhile only if its expected reuse value exceeds its write cost plus the value of the entries it evicts. The online policy approximates this oracle using reuse estimates from historical access patterns.

flowchart TD
    A[New KV state] --> B{Attention type}
    B -->|linear / sliding window| C[Host DRAM]
    B -->|full attention| D{Expected reuse > write cost + evicted value, given tier pressure?}
    D -->|yes| E[Admit to SSD]
    D -->|no| F[Skip SSD, recompute on miss]
    C -->|DRAM over capacity| D

Evidence

The reported baselines include Dynamo-style selective write-through. NVIDIA Dynamo offloads a block once its access frequency reaches 2. The paper argues that frequency-only filtering ignores state size, layer-specific reuse, tier occupancy and remaining write budget.

Figure 3 plots existing policies on hit rate against SSD lifetime, and all of them sit below 30 days of lifetime. The policies shown are LRU, LRU-K, HotPrefix, Workload-aware, Bidaw, MeritKV, RR-Evict and Marconi.

A $20k TCO comparison at equal aggregate bandwidth finds SSD outlasts an HDD cluster once each written entry is reused at least 3 times. HDD reads also count toward its rated workload, while SSD reads do not consume TBW.

The SSD lifetime model is E_TBW = C × DWPD × 365 × years, divided by WAF × workload writes. The headline 7.3× average lifetime extension appears only in the abstract and introduction of the fetched text.

Limits

The fetched text ends in Section 2.5. The evaluation setup, traces, models, hardware sizes, TTFT numbers and per-baseline results behind the 7.3× figure are not visible.

The method only helps hybrid models that mix full attention with linear or sparse attention. It relies on multi-round or agentic reuse patterns, and pure full-attention models get no attention-type signal.

The method reduces host writes only. It does not address device write amplification.

HDD integration is left for future work. The claim that the method extends to High Bandwidth Flash is stated but not demonstrated. No code, release or licence is mentioned.