DeepInfer: High-performance inference framework for DeepSeek models

DeepSeek has released DeepInfer, the high-performance inference framework that powers DeepSeek's large-scale production services. github.com's report says this open-source release provides "offline inference with the core features needed for serving at scale." The only supported model listed is DeepSeek-V4.1-Flash. The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on the first run.

This piece covers the feature list, sampler flags, accelerators, build prerequisites, benchmark hardware, and the parts a complete service still needs. It is for engineers running DeepSeek-V4.1-Flash offline on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs, and for readers following the vLLM, SGLang, and TensorRT-LLM projects named in the acknowledgments.

High-performance inference framework: what actually changed

The release packages optimized kernels with the model and sampler runtime. Parallelism is listed as Attention DP / balanced CP and MoE EP + EPLB. That same supported-feature block names DSpark and confidence-scheduled verification, SWA bounded replay, and asynchronous scheduling. The repository describes the scheduler as a "zero-overhead batch scheduler." Batch-invariant inference, deterministic sampling, and structured outputs are the other three entries. The block is the offline-inference feature set, and the only model named with it is DeepSeek-V4.1-Flash. DeepSeek's production services run on this framework.

DeepInfer is released under the Apache License 2.0, and the repository places the text in the LICENSE file. Acknowledgments say the framework was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. That note thanks their developers. The bibtex entry type is misc and the key is deepinfer2026. Its author is DeepSeek-AI, its year is 2026, and its publisher is GitHub. The title reads "DeepInfer: High-performance, large-scale inference framework for DeepSeek models." The howpublished field points at the DeepInfer GitHub repository.

High-performance inference framework: how it works

The optimized kernels are FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. DSpark and confidence-scheduled verification accelerate decoding with draft tokens. That feature uses confidence estimates to schedule verification. The documented switch is DEEPINFER_ENABLE_MTP_VARLEN=1. DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length, and the default length is 5. The repository examples contain the sampler configuration that goes with those two variables. Both variables belong to the DSpark and confidence-scheduled verification feature listed with the optimized kernels and the model and sampler runtime.

Decoder SWA bounded replay reduces decoder prefill work by replaying only a bounded suffix. Default V4.1 settings are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. The retained suffix must cover the SWA window plus the maximum backtracking length. Asynchronous scheduling overlaps CPU batch scheduling with device execution and is enabled by default. Batch-invariant inference keeps model results identical across different batch compositions when DEEPINFER_BATCH_INVARIANT=1. Deterministic sampling produces identical outputs for identical input tokens and sampling parameters. It sets DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0, and it passes a fixed seq_seed to SamplingParams. Structured outputs constrain generation to a regex, a JSON schema, or an EBNF grammar. Callers pass so_args with schema_type and schema to SamplingParams.

High-performance inference framework: why it matters now

A complete inference service, on the repository's own list, still requires three components this release does not include. The first is a KV cache system. The second is VE/Prefill/Decode disaggregation, where "most required engine functionality is already implemented." The third is load balancing across data-parallel replicas for both prefill and decode. The getting-started path included here is offline inference, launched from examples/offline_inference.py. This release provides that offline path, and it leaves the KV cache, disaggregation, and data-parallel load balancing out of the release.

Performance benchmarks are grouped under Prefill and under Decode. Prefill was measured on 4 SM100 GPUs. Decode was measured on 64 SM100 GPUs. Each of those measurements stayed within one scale-up domain. The script that reproduces the benchmarks is run_sweep.sh. Evaluation benchmarks are written up in eval/README.md. On NVIDIA, DEEPINFER_BATCH_INVARIANT=1 keeps model results identical across different batch compositions. Deterministic sampling, also NVIDIA only, produces identical outputs when the input tokens and the sampling parameters match and a fixed seq_seed is set.

High-performance inference framework: who is affected

Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Every build requires Linux, GCC >= 13.0, and Python 3.12. CUDA 13 is required for NVIDIA only. CANN 9.2 is required for Ascend only. Before building, the instructions say to create and activate an empty Python 3.12 environment. They also say to keep DEEPINFER_ACCELERATOR exported in the environment where DeepInfer runs. People who need a KV cache system, VE/Prefill/Decode disaggregation, or data-parallel load balancing still have to supply those pieces, which this release does not include.

The NVIDIA build exports DEEPINFER_ACCELERATOR=nvidia. The Ascend build sets that same variable to ascend, and the instructions say to use ascend for the Ascend platform. Either export is followed by ./build.sh. Offline inference then downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 into the Hugging Face cache on first run. A local copy is chosen by setting DEEPINFER_HF_MODEL_DIR to /path/to/DeepSeek-V4.1-Flash-1017. The command is python examples/offline_inference.py. Build notes tell operators to keep DEEPINFER_ACCELERATOR exported wherever DeepInfer runs, and they list Linux for both the NVIDIA and Ascend paths.

High-performance inference framework: what to watch

Batch-invariant inference and deterministic sampling are both marked NVIDIA only. Deterministic sampling means identical outputs for identical input tokens and sampling parameters. It requires DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0, and a fixed seq_seed on SamplingParams. DSpark documents the other value, DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE defaults to a maximum draft length of 5. Those two settings of DEEPINFER_ENABLE_MTP_VARLEN are opposites. Ascend 950 is specified with DEEPINFER_ACCELERATOR=ascend and CANN 9.2, apart from the NVIDIA-only pair. Batch-invariant mode keeps model results identical across different batch compositions, and the repository marks that mode NVIDIA only.

Default V4.1 SWA replay keeps DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE at compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS at 140, and the retained suffix must still cover the SWA window plus the maximum backtracking length. Asynchronous scheduling stays enabled by default, overlapping CPU batch scheduling with device execution. Structured outputs still require so_args schema_type and schema for a regex, a JSON schema, or an EBNF grammar. A complete service still needs the missing KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode.