[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback

What it claims

DeepSeek released DeepInfer on 20 October 2026 under Apache 2.0. It is the vLLM-derived inference engine behind DeepSeek V4.1 Flash, released together with deepseek-recipe, a Rust library with Python bindings. The repository supports distributed inference, speculative decoding, SWA Bounded Replay, structured output, batch-invariant inference and deterministic sampling.

DeepSeek reports these production figures: 400 ms average time to first token, 230 tokens/s generation (300 tokens/s for coding), and 96% input-cache hits. All production figures are first-party and unaudited.

"Fully open source" is narrower than it sounds. KV-cache management, prefill/decode disaggregation and DP replica load balancing are not released, because they remain coupled to DeepSeek's internal infrastructure.

How it works

• Stage separation. Vision encoding, prefill and decode scale separately. The recommended shapes are 8-accelerator machines for prefill and 64 to 128-accelerator supernodes for decode, with wide expert parallelism. Commentator @nrehiew_ notes that the disclosed Blackwell configuration appears to use no tensor parallelism.
• Prefill scheduling. Work is allocated by estimated attention compute, not by request length. Chunk sizes vary to fit a time budget. Because prefixes live in shared storage, requests need no cache affinity and go to any free worker.
• Decode routing. Routing balances running-request count and accumulated KV length.
• Expert load balancing. Popular experts are replicated, which trades HBM for throughput with diminishing returns.
• Asynchronous CPU scheduling. Host-side scheduling overlaps accelerator execution, which removes gaps between decode steps.
• Persistent cache. Block KV sits in a roughly 100 PiB SSD 3FS cluster. Agent traffic is retained 3 days and chatbot traffic 7 days. Loads overlap layer execution and use GDR/RDMA, because fine-grained copies through the CUDA copy engine perform poorly.
• Short-lived state. Sliding-window attention state stays in host memory. When it is missing, SWA Bounded Replay rebuilds an approximation from only the latest 128 tokens and reuses the block cache. DeepSeek says this nearly halves total prefill compute.
• DSpark. DSpark predicts draft-token acceptance and adapts verification length. It gives 6% more throughput than always verifying 5 draft tokens. This baseline is a fixed speculative setup, not plain decoding.
• deepseek-recipe. It serves the Chat Completions, Responses and Messages APIs and parallelizes tokenization. It distinguishes real control tokens from identical strings in user text. The page says this is not a full prompt-injection defense.

flowchart TD
  R[Request] --> P[Any free prefill worker]
  P --> C{Block prefix in 3FS?}
  C -- yes --> L[Async load via RDMA]
  C -- no --> F[Full prefill]
  L --> S{SWA state in host RAM?}
  S -- yes --> D[Decode supernode]
  S -- no --> B[Bounded Replay, last 128 tokens]
  B --> D
  F --> D

Evidence

The vLLM team reports decode throughput within 6% of DeepInfer's published numbers on GB200 under a matched configuration, and it published reproduction recipes. This shows decode-stage parity only. It does not cover cache operations, routing or end-to-end latency, and the competing team reported it themselves.

Inference engineer Su Jin Yan reports that the long retention tail adds only about 0.5 to 1 point of cache hits.

The margin figures are estimates by commentators, not disclosures by DeepSeek. DeepSeek disclosed revenue of $38 per accelerator-hour at off-peak prices. At $5 to $10 per hour in GPU cost, that gives roughly 73.7 to 86.8% before SSD, CPU, networking, idle capacity and staff costs. @zephyr_z9 first estimated 87.76 to 96.94%, then corrected it to 84.2 to 96.1%. Yuchen Jin notes that DeepSeek did not name the GPU behind the $38 figure.

The 155T tokens/day figure counts cached input tokens, so it is not a measure of newly computed work.

How to use it

The build targets NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. It needs Linux, Python 3.12, and either CUDA 13 or CANN 9.2. You choose the platform with an environment variable before building:

DEEPINFER_ACCELERATOR=nvidia   # or ascend
./build.sh

The repository describes itself as an offline inference framework. Online serving needs external KV-cache management, P/D disaggregation and load balancing. Teortaxes suggests 3FS plus possibly Dynamo could fill those gaps, but no one has shown this working. Teortaxes estimates that matching DeepSeek's cluster throughput would take roughly 3,000 to 4,000 GB200s. DeepSeek says future model implementations will land in the same repository. An official English technical report is in the repository.

Limits

• Model specificity. The gains are tied to V4.1 Flash. Its Engram parameters, compressed KV and the approximation in Bounded Replay may not transfer to other checkpoints. No one has published portability results.
• Hardware. Only next-generation SM100 is listed on the NVIDIA side, not widely deployed GPUs. Ascend is supported, but its production efficiency is unmeasured. Teortaxes's figures put GB200 at about 5 to 7.4 times Ascend 950DT decode throughput, but these mix different measurements and assumed prices.
• Value of the release. Su Jin Yan judges the engineering report more useful than the code for experienced inference developers, because key components are missing.
• Source coverage. This analysis covers only the first 40,000 of 56,439 characters of the page.