Cache Engineering for Agents

What it claims

Cache engineering adds two KV cache operations for agents, drop and repos. Because the trajectory text never changes, retained KV states are reused without re-prefill. The motivating cost figure: KV cache reads make up 82.4% of total API cost in the authors' measurement. A recursive self-improvement (RSI) search over cache policies cuts cost by a further 7.1% on held-out tasks compared with the expert-designed policy. The paper is preprint v1 from Tsinghua University and CUHK, dated 20 October 2026. Code is at https://github.com/thu-nics/AKV.

How it works

The paper defines three operations:

• append is standard KV caching. The position index starts at 0 and advances by one per token.
• drop(P) removes the listed logical token indices from future attention. Retained entries keep their position encodings, and the next position index is not reset. Dropping alone can therefore still push positions past the model's context limit.
• repos renumbers retained keys to positions 0 to r−1 and sets the next index to r. For RoPE it rotates each cached key from position p to q with K ← R(q)R(p)⁻¹K. Values are unchanged.

Retained entries still carry information from dropped history, because they were computed while attending to it. This makes cache state history-dependent. The same tokens and retained positions can yield different KV depending on the order of drops and appends. As a result, plain prefix matching (as in vLLM or SGLang radix caching) cannot reuse these states correctly.

AKV handles this with three parts:

• A stateless, API-compatible interface. The client resends messages plus cache instructions and does not track what is resident on the server.
• An extended radix tree with "drop nodes," which branch on when a drop happened.
• Segment-batch attention, which rebuilds missing states in one prefill pass.

The example policy is tool-response rolling drop. It keeps all assistant reasoning and the K most recent tool responses, and drops older tool-response cache. The authors justify it by attention measurements: tool responses get less per-token attention than assistant messages at the same relative position, and older tokens get less than recent ones.

Evidence

The motivating experiment uses GPT-OSS-120B on the first 100 BrowseComp-Plus (BCP) tasks, with 4 trials per task and at most 100 rounds.

The paper reports AKV results at K = 1, 4 and 12 retained tool rounds. The headline results span three LLMs and four agent benchmarks, measured against threshold-triggered summarization. The RSI search used MiniMax-M2.7 on BCP and reached 15.6% and 17.8% lower per-task cost than the human-designed policy at points in the search.

How to use it

Requests carry a cache field next to standard messages (illustrative, from the paper's Figure 6):

{ "model": "gpt-oss-120b",
"messages": [
{"role": "user", "content": "Search ICLR"},
{"role": "assistant", "content": "Searching"},
{"role": "tool", "content": "ICLR is..."}
],"cache": {
"drop": {"1": [0],
"2": [1]},
"reposition": [2]
} }

The figure annotates these entries as drop m0, drop m1, then repos. The fetched text gives no install commands, model support list or licence.

Limits

The fetched text stops in Section 3. The per-benchmark tables, the names of the other benchmarks and models, the ablations and the hardware setup are not visible. repos is specified only for RoPE models. Dropping is not equivalent to deleting text, since retained KV still encodes the dropped content, so behaviour differs from trimming. The throughput and latency gains require running AKV as the serving engine.