DeepInfer: High-performance inference framework for DeepSeek models

What it claims

DeepInfer is the inference framework behind DeepSeek's production services, released as an offline-inference subset under Apache 2.0. It supports one model, DeepSeek-V4.1-Flash; the default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017. Its kernels come from FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. The README says it was originally built mainly on vLLM, with ideas taken from SGLang and TensorRT-LLM.

How it works

Most features are switched on with environment variables:

• DSpark draft decoding with confidence-scheduled verification: set DEEPINFER_ENABLE_MTP_VARLEN=1. DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, which defaults to 5.
• Decoder SWA bounded replay: this cuts decoder prefill work by replaying only a bounded suffix of the input. The default V4.1 settings are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. The kept suffix must cover the sliding-window size plus the maximum backtracking length.
• Asynchronous scheduling: CPU batch scheduling overlaps with device execution. It is on by default.
• Batch-invariant inference: set DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
• Deterministic sampling: set DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0, then pass a fixed seq_seed to SamplingParams (NVIDIA only). Deterministic sampling therefore turns off draft decoding.
• Structured outputs: pass so_args={"schema_type": ..., "schema": ...} to SamplingParams. Regex, JSON schema and EBNF grammars are supported.

Parallelism uses attention data parallelism or balanced context parallelism, plus MoE expert parallelism with EPLB.

How to use it

Requirements are Linux, GCC 13.0 or later, and Python 3.12 in a new, empty environment. NVIDIA also needs CUDA 13. Ascend also needs CANN 9.2.

export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Use ascend as the accelerator value on Ascend NPUs. The variable must stay exported in the shell that runs DeepInfer. On first run the checkpoint downloads into the Hugging Face cache. To use a local copy, set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017.

python examples/offline_inference.py

The benchmark sweep script is run_sweep.sh. Evaluation instructions are in eval/README.md.

Limits

The page text gives the benchmark setup but no figures: 4 SM100 GPUs for prefill and 64 for decode, each group inside one scale-up domain. That means the reported configuration needs a 64-GPU decode deployment. Batch invariance and deterministic sampling are NVIDIA-only.

A full serving stack still needs three things the release leaves out: a KV cache system, VE/prefill/decode disaggregation, and load balancing across data-parallel replicas. For disaggregation, the README says most of the engine functionality needed is already implemented.