Skip to content

Back to the catalog

Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth Flash
arxiv.orgpaper

Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth Flash

Lachesis is a lifetime-aware KV cache placement layer for LLM agent serving that sits between the agent harness and the serving engine, deciding whether to write each KV cache segment to HBM or high-bandwidth flash (HBF) based on its expected lifetime. The authors identify temporal, structural, and inter-worker axes along which KV cache lifetime diverges in agentic workloads, and show in trace-driven simulation that lifetime-aware placement extends HBF's write-endurance-limited lifetime by 1.19–3.13x over naive HBM-first placement, reaching 3.3–12.2 device-years. This matters for serving infrastructure as agentic workloads with long-lived, multi-agent context accumulation push memory capacity demands beyond what HBM alone can economically provide.

TL;DR A lifetime-aware KV cache placement layer extends high-bandwidth flash's write endurance by up to 3x for agentic LLM serving workloads.
Open original ↗
Document

What it claims

The paper argues that KV cache should be split between HBM and HBF by expected lifetime, not by read-to-write ratio as in prior work (Kyung et al.). HBF wears out with writes. HBM absorbs more writes when its blocks are reclaimed often. So putting short-lived segments in HBM and freeing them at their discard point keeps writes off flash. The harness decides when context is discarded, so lifetime can be known or predicted before the KV is written. The read-to-write criterion assumes stateless requests where all KV shares one lifetime. That assumption breaks for stateful agent sessions.

How it works

The unit of placement is a context segment: a group of tokens the harness discards together at a context boundary. Segment lifetime differs along three axes:

  • Temporal (turns). At the next user message, the harness strips the turn's reasoning. Reasoning is 66–83% of the KV a turn writes but lives one turn. The surviving output lives 7.6–17.2× longer.
  • Structural (delegation). On sub-agent return, the harness keeps only the result. Sub-agent segments are most of a turn's KV writes. The lead-agent segment lives 4.0–6.1× longer.
  • Inter-worker. Workers of the same class finish at different times, with a p10–p90 lifetime spread of 3.7–23.6×. Their relative order is predicted at spawn time from the record the harness uses to build each worker's context.
flowchart TD
  S[New segment write] --> Q{Boundary known to harness?}
  Q -->|yes: temporal/structural| D{Discarded at boundary?}
  D -->|short-lived| H[HBM priority]
  D -->|long-lived| F[HBF]
  Q -->|no: inter-worker| P[Predict lifetime at spawn]
  P --> R[Rank shortest-first]
  R -->|top-k fits HBM| H
  R -->|rest| F
  H --> X[Free at boundary]

Engine interface. Lachesis tracks lifetime entirely on its own. The serving engine sees it through only two calls:

  1. Allocate a KV block from a specified tier (HBM or HBF).
  2. Free a segment's blocks at the boundary the harness reports.

Evidence

  • Baseline. The comparison is HBM-first placement: allocate in HBM while a free block exists, otherwise HBF. Both policies write the same total KV volume; only the tier split changes.
  • Headline result. In trace-driven simulation of a two-tier memory system, HBF lifetime improves 1.19–3.13×, reaching 3.3–12.2 device-years.
  • Warranty claim. Under 24×7 operation at the full load a tight SLO admits, HBF outlasts a five-year warranty on the multi-agent trace only. The text does not claim this for the other traces.
  • Characterization setup. All workloads use Gemma 4.
  • The temporal regime uses the Harbor harness on τ³-bench and BFCLv4 multi-turn.
  • The structural and inter-worker regimes use AOrchestra, which inherits from Claude Code, on AppWorld, DeepSearchQA, SWE-bench Verified, TerminalBench and WebArena.
  • Motivation figure. Agentic peak contexts are 29× those of chatbot workloads, for example 37K–69K tokens versus 0.5K–2.8K.

Limits

  • Simulation only. Results come from trace replay on a simulated memory system, not real HBF hardware.
  • Performance is out of scope. HBF read latency and plane-parallel layouts are deferred to prior work.
  • Harness integration required. The harness must expose and report context boundaries.
  • Narrow characterization. Each regime was configured so that exactly one boundary is active, and only one model (Gemma 4) was characterized.
  • Truncated page. The fetched text ends partway through §3. The predictor's design, per-trace results, the breakdown of the 1.19–3.13× range and any ablations were not available. Check §4–5 before relying on the predictive path.

Technologies named here

Techniques named here

  • Lifetime-aware KV cache placement

Related in inference