Efficient KV Cache Migration for Geo-Distributed LLM Inference in Collaborative Edge Computing

What it claims

EdgeFlow migrates the KV cache of a model shard off an overloaded or departing edge server. It splits each shard's layers into two groups. For some layers it sends the layer's input activations and the target recomputes K and V. For the rest it sends the KV cache directly. Activation transfer and recomputation are pipelined.

Across multiple overloaded servers, choosing target nodes and the split ratio is posed as the MN-KVM problem, shown NP-hard. The authors give a 2-approximation algorithm with O(log K + log T_max) complexity, where K is the number of overloaded servers. They report up to 45% lower migration time than direct KV transmission and token-based recomputation.

How it works

Cost model. For one layer with batch B, sequence length L_seq, activation width H_j and link bandwidth B_k,n:

• Direct KV transfer costs 2·L_seq·H_j·B / B_k,n, because it sends both K and V.
• Activation transfer costs half that.
• Recomputation adds 2·L_seq·H_j²·B / FLOPS_n on the target, applying K = W_k·H and V = W_v·H.

The trade-off is half the bytes in exchange for target-side compute. The idea only works if communication and compute can overlap across layers.

Why a hybrid beats pure activation transfer.

• On weak targets, recomputation becomes the bottleneck.
• Mismatched transfer and compute times leave pipeline bubbles.

Sending some layers as raw KV fills those bubbles.

Two results reduce the search space:

• Theorem 1: with the layer split fixed, sending all activations first is optimal.
• Theorem 2: only the number of activation-layers matters, not which layers. So the decision per shard is one integer, p (the first p layers are recomputed).

flowchart LR
  P[Profile: bandwidth, GPU perf, free mem, batch, seq len, layers] --> S[Scheduler: target node + p]
  S --> A[Layers 1..p: send activations]
  S --> K[Layers p+1..N: send KV cache]
  A --> R[Target recomputes K,V]
  R --> T[Target restored]
  K --> T

Workflow:

1. Collect node availability, inter-node bandwidth, GPU type, utilisation and memory, plus the user-supplied model configuration.
2. The scheduler solves MN-KVM using decomposition, a modified LP relaxation and bipartite-graph construction.
3. Execute the migration.

Evidence

Experiments ran on a real geo-distributed inference platform of "dozen[s] of heterogeneous GPUs from several major universities." The baselines are direct KV transmission and pure KV recomputation. The only result in the fetched text is the headline "up to 45%". Per-model, per-bandwidth or per-GPU breakdowns are not included.

Limits

• Truncated text. The fetched text stops mid-Section III. It contains no experimental setup, no models tested, no results tables and no ablations. "Up to 45%" is a best case whose conditions you cannot see from this page.
• Simplified cost equations. By the authors' own note, they omit residual connections, layer norm and positional encodings. Positional encodings matter for recomputing K under rotary-style schemes.
• The 50% saving depends on an attention assumption. The cost model assumes multi-head attention, where K and V each match the activation width H_j. Check how the saving holds for models whose KV is narrower than the activations.
• Weights must already be on the target. Recomputation requires the target to hold that shard's W_k and W_v.
• No code release. No code, licence or artifact is mentioned. The scheme is described within the authors' platform; the paper cites Petals and PolyLink as related systems.