KV-streams for Efficient Compaction in Agentic Reinforcement Learning

What it claims

KV-streams removes the repeated prefill that context compaction normally forces during agentic RL. Instead of restarting a fresh trace after each compaction, the inference engine evicts KV entries in place and keeps one continuous cache. The trainer reproduces the eviction with an attention mask over a single trace. Reported speedups are a 5× wall-clock speedup on TextWorld, 1.3× on ALFWorld, and reaching peak SWE-bench Verified performance 3× faster than full context. The abstract quotes 2.6–5× across strategies, which does not match the ALFWorld figure. The paper also claims three further results:

• With enough training, KV-streams closes and can exceed the gap to full context.
• It generalizes better to unseen text games.
• The "pseudo-recurrent" KV state emerges from RL alone. This contradicts Kontonis et al. (2026), who needed an SFT stage first.

How it works

A compaction operator κ = gen ∘ del fires on a trigger: either a token/turn budget B or the model's own decision. del removes a span, and gen optionally emits a bridge such as a summary. The two pipelines differ in what happens next:

• Re-prefill compaction: the r retained entries (overlap o plus summary s) are re-emitted with reset positions, so their KVs must be recomputed. Over an N-entry rollout this costs about N·r/(B−r) extra prefilled entries. That cost diverges as r approaches B.
• KV-streams: the retained entries keep their original (k, v). The extra prefill cost is 0.

flowchart TD
  T[Budget or model trigger] --> K["κ: delete span, gen bridge"]
  K --> RP[Re-prefill: new trace, reset positions]
  K --> KS[KV-streams: evict in engine, keep KVs]
  RP --> C1[Retained tokens prefilled again, split traces]
  KS --> C2[One trace; trainer attention mask hides deleted span]

The paper tests four compaction strategies, ordered by how much context they retain:

Evidence

All three settings use 8 rollouts per prompt and a learning rate of 1e-6. On SWE, only Markovian Thinker and Sliding Window were compared against full context. Runs were stopped once their wall-clock time exceeded that of the full-context run. On the SFT question, the paper reports that adding an SFT stage cost wall-clock time and did not improve performance. Synthetic experiments show the pseudo-recurrence appearing without SFT.

How to use it

Code is at github.com/Emilianopp/KV-streams, with an SGLang fork at github.com/Emilianopp/sglang. The authors say only a subset of the experiments is released. KV-streams requires changes in both the inference engine and the trainer:

• Text games: forked vLLM plus forked prime-RL, async with max off-policy lag 3.
• vLLM's 16-token prefix-cache blocks leave trailing tokens uncommitted, which broke the stream and "substantially" hurt performance. The fix is padding every completion to a multiple of 16 tokens, which inflates token counts. The paper accounts for this padding in its figures.
• In-flight weight updates must not flush the KV cache. Flushing would force full-trace prefills and undo the savings.
• SWE: forked Slime trainer plus forked SGLang. SGLang commits tokens with block size 1, so no padding is needed.

Limits

• Seeds: only ALFWorld is multi-seed. TextWorld is single-seed; the SWE description in §4 also says single-seed, while the Figure 1 caption reports three seeds on SWE-bench Verified.
• Early training: both KV-streams and re-prefill compaction can underperform full context early in training. KV-streams closes the gap only with sufficient training.
• SWE coverage: only two strategies were tested on SWE, at a single 4B model scale.
• Recurrence claim: the "RL alone" recurrence result comes from a controlled synthetic setting.
• Speedup varies: the gain depends heavily on how much context the strategy retains, from 1.3× on ALFWorld to 5× on TextWorld.
• Inconsistent counts: the paper says three strategies in the abstract and four in the body.