Cache Engineering for Agents (AKV)

What it claims

AKV lets an LLM agent edit its own active KV cache instead of its transcript. The authors argue this avoids the re-prefill cost that context engineering (trimming or summarizing history) pays. A tool-response rolling-drop policy scores 1.7 points higher on average and costs 27.7% less than threshold-triggered summarization. Running that policy on the AKV serving engine gives up to 1.2x output throughput and a 1.5x speedup in P95 request latency. Letting the system recursively improve its own cache policies cuts cost by a further 7.1% over an expert-designed policy.

How it works

The motivating problem is that changing any part of a trajectory prefix invalidates every cached state after it. The retained context then has to be re-prefilled. Agents that limit re-prefill by keeping less context can lose task accuracy.

Cache engineering leaves the token trajectory intact and exposes two operations to the agent:

• drop excludes selected cache entries from future attention.
• repos adjusts the position encodings of the keys that remain.

flowchart TD
    A[Trajectory grows] --> B{How to shrink context?}
    B -->|Context engineering| C[Trim or summarize prefix]
    C --> D[Later cache invalidated]
    D --> E[Re-prefill retained context]
    B -->|Cache engineering| F[drop / repos on KV cache]
    F --> G[Trajectory unchanged, no re-prefill]
    G --> H[History-dependent cache state]
    H --> I[AKV engine tracks op sequence]

The authors identify "history dependence" as the main serving problem. The same token sequence can produce different cache states depending on which cache operations were applied to it. Standard prefix matching therefore cannot decide correctly when a cache can be reused. The AKV engine handles this by keeping a stateless interface that is compatible with existing serving APIs. Internally, it manages stateful cache reuse, eviction and restoration.

How to use it

Code is at https://github.com/thu-nics/AKV. The page gives no install commands, supported models or licence. The Hugging Face page lists no models, datasets or Spaces linked to the paper. The paper text can be fetched with the Hugging Face CLI:

hf papers read 2610.28577

Limits

The page shows only the abstract. It does not name the benchmarks, base models, hardware or context lengths behind the 1.7-point, 27.7%, 1.2x and 1.5x figures. The throughput and latency gains are stated without naming the baseline engine. The abstract does not say how the agent decides what to drop, beyond the rolling-drop policy for tool responses. It also does not describe the recursive self-improvement procedure.