DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

What it claims

DualPace gives CPU-resident decode requests their own scheduler, queues and clock, instead of letting them advance only when the GPU scheduler finds an overlap window. The authors call the older pattern "GPU-gated CPU offloading" and identify it as the cause of bursty CPU-side progress and inflated P99 inter-token latency (ITL). Besides the throughput gains in the summary, DualPace keeps NEO's P99 time-to-first-token advantage over vLLM. It also sustains 7.3–50.5% higher request rates than NEO at the same P99 ITL.

How it works

NEO splits each iteration into two sub-batches. It overlaps CPU attention for one sub-batch with GPU linear work for the other, and admits a CPU decode only if two conditions hold:

• T_lin0 ≥ T_ca1
• T_lin1 + T_ga0 ≥ T_ca0

It also requires the pipeline's predicted request rate to beat GPU-only execution. Decodes contribute one token row each to GPU linear work, so GPU linear time shrinks when prefills are scarce. CPU attention time grows with KV length. When prefills drain, the conditions fail and CPU-resident progress stalls.

DualPace uses a shared LLMEngine with two independently scheduled engines, each with its own scheduler and worker group. The GPU engine owns prefills and GPU-resident decodes. The CPU engine batches decodes over host-resident KV using host PagedAttention and packed oneDNN BF16 kernels on Intel AMX. For each CPU batch, offline hardware profiles decide whether dense stages run on the CPU or go to the GPU through "Dense Assist". Either way, the request stays under CPU scheduling control.

flowchart LR
  E[Shared LLMEngine] --> G[GPU engine: prefill + GPU decodes]
  E --> C[CPU engine: own queue and clock]
  C --> P{Offline profile threshold}
  P -->|CPU efficient| L[Full decode on CPU via AMX]
  P -->|GPU preferable| D[GPU Dense Assist for dense stages]
  D --> G

Evidence

The headline trace runs Llama-3.1-8B on an RTX 4090 with 12 physical Intel Xeon Silver 4510 (Sapphire Rapids) cores. It serves 2,000 Azure-Conversation requests. In this trace, NEO's CPU-resident token count shows multi-second stalls and then plateaus after about 600 s, while its total token count keeps rising. DualPace advances CPU-resident tokens continuously and finishes the trace first. NEO's P99 ITL climbs steeply across request rates of 2 to 4.5 req/s.

Operator microbenchmarks on the same hardware show the following:

• Attention: CPU PagedAttention matches the GPU at batch size 1. It stays within about one order of magnitude at batch sizes 32 and 128.
• Dense projections: For QKV and FFN gate/up projections, AVX512 trails the RTX 4090 by nearly two orders of magnitude. AMX narrows the gap to about one order, for 1 to 1,024 tokens.

The paper also cites prior work (LIA). It reports AMX reaching 20 TFLOPS on Sapphire Rapids and 40 TFLOPS on Granite Rapids.

Limits

The fetched text stops at the design section (§4.2), so the scheduling policy, routing thresholds and full evaluation matrix are not visible here. The throughput gains are stated only for "decode-oriented scenarios". The CPU path depends on Intel AMX. Its dense-operator advantage over AVX512 is the reason full CPU decode is practical at all.

The authors note that CPU batches beyond about 1,024 requests directly lengthen token iterations. Routing thresholds come from offline profiling of each hardware setup, so they must be re-derived on new machines. The page mentions no code release, licence or run instructions.