SARIA: Persistence-Aware KV Caching with Coordinated DRAM--SSD Reload for LLM Agents

What it claims

SARIA is a DRAM and SSD KV cache layer for multi-turn agents, built on LMCache MP. It targets two problems the authors measure on standard GPU servers: SSD wear from writing every KV block, and too few SSDs to match GPU ingress bandwidth. Persisting every block can wear out a 1-DWPD drive in weeks. For GLM-5.2 on B300 GPUs (12,963 prefill tokens/s per GPU at 8K inputs), full duplicate caching writes about 1,145 MiB/s, or 103.7 TB/day, per GPU. That exhausts a 4-TB 1-DWPD SSD's five-year budget in about 70 days. Buying bigger SSDs buys little: on 350 TraceLab sessions, raising capacity from 3M to 32M token slots lifts hit rate only from 82.0% to 84.8%.

How it works

flowchart LR
  E[Block evicted from DRAM] --> T{Evicted repeatedly or reuse count high?}
  T -- yes --> S[Persist to SSD]
  T -- no --> D[Drop, no SSD write]
  S --> G[GDS read to GPU]
  R[DRAM replica] --> C[Coordinator paces DRAM reads]
  G --> C
  C --> P[Prefill]
  P --> L[Lazy partial promotion HBM to DRAM]
  L --> R

• Write path. A block goes to SSD only after repeated DRAM evictions, which the authors treat as evidence of reuse beyond DRAM residency. Very popular blocks, judged by reuse count, are persisted before eviction. Both thresholds are user-configurable, which trades SSD write volume against cache coverage.
• Hybrid models. Unified context metadata covers full-attention and hybrid models. Hybrid models need a recurrent-state checkpoint at the exact prefix boundary, so reload planning finds valid prefixes across both tiers.
• Promotion. SSD hits load straight to GPU through GPUDirect Storage. After prefill, some of those blocks are copied from HBM into DRAM. The promotion ratio is chosen so DRAM and SSD together fill GPU ingress without crowding DRAM. This sits between zero promotion (FlexKV and Dynamo KVBM GDS paths) and full promotion (HiCache).
• Coordination. The coordinator throttles DRAM reads while GDS is active, so the SSD transfer does not leave a long tail. Full DRAM bandwidth returns once the SSD transfer ends. Leftover ingress serves DRAM reloads for other requests.

Evidence

Table 1 estimates per-H200 SSD endurance under full duplicate caching, assuming a 4-TB, 1-DWPD drive:

The motivation study uses 350 sessions averaging 18.4 requests and 83.7K prompt tokens per request. Session starts are compressed into 12-, 4- or 2-hour windows to raise load. DRAM is 100 GB, about 2.2M GLM-5.2 token slots.

• DRAM coverage. At 12 hours, 79.3% of cache-hit requests are served entirely from DRAM. At 2 hours this falls to 52.4%.
• Unread SSD writes. 82.5% of SSD writes are never read at 12 hours, 65.0% at 4 hours and 44.3% at 2 hours.
• Promotion at 2 hours. Half promotion beats full promotion on tail transfer time: P5 is 277 ms versus 310 ms, P10 is 231 ms versus 252 ms, and the mean is 78 ms versus 81 ms. Zero promotion has a 105 ms mean and a 390 ms P5.
• Promotion and eviction. More promotion raises DRAM evictions, starting from 59,589 with zero promotion. The fetched text breaks off at the comparison figure of 97,097.

End-to-end results come from TraceLab and WEKA workloads on full-attention and hybrid models, with one or two SSDs. SSD writes fall 57.1 to 58.9% against full duplicate caching. Mean TTFT falls 42.3 to 79.7% against DRAM-only caching. In the ablation, partial promotion, tier overlap and coordination together cut reload time by 26.7 to 31.9% for the slowest quartile.

How to use it

The page states that SARIA is built on LMCache MP and uses GPUDirect Storage. It gives no repository, release or licence.

Limits

• Truncated text. The fetched text ends in Section 3.2. Per-configuration end-to-end tables, GPU and SSD hardware for the main runs, and the threshold settings behind the 57.1 to 58.9% figure are not shown.
• Hit-rate trade-off. Figure 1 states that SARIA extends SSD lifetime by giving up some cache hit rate.
• Narrow SSD setups. Only single- and dual-SSD configurations are evaluated.
• Simulated promotion study. The promotion comparison assumes ideal DRAM and SSD overlap, with GDS bandwidth set to half of GPU ingress.