[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback

What it claims

DeepSeek open-sourced DeepInfer, the inference engine behind V4.1 Flash, under Apache 2.0, together with deepseek-recipe, a Rust library with Python bindings. DeepInfer began as a customization of vLLM. DeepSeek's engineering report gives these first-party production figures: 155T tokens/day from one cluster, 96% input-cache hits, 400 ms average time to first token, and 230 tokens/s generation (300 tokens/s for coding). These figures are not audited. The release does not reproduce the full service. KV-cache management, prefill/decode disaggregation and DP replica load balancing are not fully released, because they are coupled to DeepSeek's internal infrastructure.

How it works

Vision encoding, prefill and decode run as separate stages. The recommended shapes are 8-accelerator machines for prefill and 64–128-accelerator supernodes for decode, with wide expert parallelism.

• Prefill allocates work by estimated attention compute, not request length, and varies chunk size to fit a time budget. It needs no cache affinity, because prefixes sit in shared storage.
• Decode routing balances running-request count against accumulated KV length.
• EPLB replicates popular experts and redistributes them periodically. This trades HBM for throughput, with diminishing returns.
• Async CPU scheduling removes idle gaps between decode steps.
• DSpark predicts draft-token acceptance and adapts verification length. It is 6% faster in production than always verifying 5 draft tokens. That is a gain over that baseline, not over non-speculative decoding.

flowchart TD
  R[Request] --> P[Any prefill worker]
  F[3FS block KV, ~100 PiB SSD] --> P
  H{SWA state in host memory?}
  P --> H
  H -- yes --> D[Decode]
  H -- no --> B[Bounded Replay: last 128 tokens]
  B --> D

Block KV persists on a 3FS cluster of about 100 PiB SSD. It is kept 3 days for agent traffic and 7 days for chatbot traffic, and loaded asynchronously over GDR/RDMA rather than the CUDA copy engine. Sliding-window state stays in host memory. When it is missing, SWA Bounded Replay rebuilds it from only the latest 128 tokens. This is an approximation, and DeepSeek says it nearly halves total prefill compute. Engineer Su Jin Yan reports that the long retention tail adds only about 0.5–1 point of cache hits.

deepseek-recipe handles Chat Completions, Responses and Messages APIs, parallelizes tokenization, and separates real control tokens from identical-looking user text.

Evidence

The vLLM team reports decode throughput within 6% of DeepInfer's published GB200 results under a matched configuration, and published reproduction recipes. That is decode parity only. It says nothing about cache operations, routing or end-to-end latency, and the vLLM team measured it themselves.

A hardware comparison by Teortaxes puts decode at 131K context at about 25.5K (GB200) versus 5K (Ascend 950DT) tokens/s/device at 100 TPS/user, and 20K versus 2.7K at 230 TPS/user.

On margins, $38 per accelerator-hour is revenue at off-peak prices, and DeepSeek did not name the GPU. Against $5–10/hour hardware cost, the accelerator-only margin is about 73.7–86.8%. zephyr_z9's widely shared 87.76–96.94% estimate was corrected to 84.2–96.1%. All of these figures leave out the SSD fleet, CPUs, networking, idle capacity and staff, and they assume a cache-heavy token mix. The 155T tokens/day counts cache hits and is not newly computed tokens.

How to use it

DeepInfer targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It requires Linux, Python 3.12, and either CUDA 13 or CANN 9.2. You select the platform at build time:

DEEPINFER_ACCELERATOR=nvidia|ascend
./build.sh

Online serving needs your own KV-cache store, PD disaggregation and DP load balancing. Teortaxes suggests 3FS plus possibly Dynamo could fill those gaps, but that path is not demonstrated. An official English technical report is in the repository.

Limits

• Model-specific. The gains depend on V4.1 Flash co-design, including Engram, compressed KV and Bounded Replay. Nobody has shown they transfer to other checkpoints.
• Hardware. Ascend is supported, but Ascend production efficiency is not shown. The disclosed Blackwell configuration uses no tensor parallelism.
• Scale. Su Jin Yan argues wide-EP prefill pays off only at high traffic and favors pipeline parallelism for small deployments. Su Jin Yan also judges the engineering report more useful than the repository.
• Consumer GPUs. Commenters estimate about 3 tokens/s for V4.1 Flash on 12GB VRAM plus 64GB RAM.