DeepInfer: DeepSeek's datacenter inference engine

DeepInfer is DeepSeek’s in-house inference framework, open-sourced October 2026 under the Apache 2.0 license. It powers the production serving infrastructure behind chat.deepseek.com, processing 155 trillion tokens per day. It is not a general-purpose serving engine — it is a specialized, single-model optimizer for DeepSeek V4.1-Flash on NVIDIA SM100 datacenter GPUs.

​

What DeepInfer is

DeepInfer is an offline inference framework — it ships no API server, no KV cache persistence, no prefill/decode disaggregation, and no load balancer. It is the compute kernel layer that sits beneath a serving stack, not a replacement for vLLM or SGLang.

​

Hardware requirements

DeepInfer’s SM100 requirement is the most important thing to understand before evaluating it. The SM100/SM120 split is the same architecture family (Blackwell) but different silicon. Consumer Blackwell (SM120/121) uses different tensor core instructions that DeepInfer does not target. System requirements:

• Linux (Ubuntu 22.04+ recommended)
• GCC >= 13.0
• Python 3.12
• CUDA 13 (NVIDIA) or CANN 9.2 (Huawei Ascend)

​

Performance

DeepInfer’s benchmarks come from a 4-GPU prefill / 64-GPU decode cluster running V4.1-Flash: These numbers require the full 64-GPU decode cluster. A 4-GPU setup delivers substantially less.

​

Key techniques

DeepInfer combines several inference optimizations:

• FlashMLA — DeepSeek’s custom MLA (Multi-head Latent Attention) kernel for the compressed KV cache
• DeepGEMM — Custom GEMM kernels for MoE expert computation
• Attention DP + MoE EP — Data parallelism for attention, expert parallelism for MoE layers
• DSpark verification — Confidence-scheduled speculative verification using the attached DSpark head
• SWA bounded replay — Sliding-window attention with bounded replay buffer
• Asynchronous scheduling — Overlapping compute and memory operations
• Batch-invariant inference — Deterministic sampling regardless of batch size
• Structured outputs — Regex and JSON schema constrained generation

​

DeepInfer vs vLLM vs SGLang

DeepInfer is built on vLLM internals and cites SGLang and TensorRT-LLM as inspiration. For general-purpose serving, vLLM or SGLang are the practical choice. DeepInfer is relevant only if you run V4.1-Flash on SM100 or Ascend 950 at datacenter scale.

​

Should you use DeepInfer?

Use DeepInfer if:

• You operate 64+ SM100 GPUs (B200, GB200) or Ascend 950 NPUs
• You serve DeepSeek V4.1-Flash as your primary model
• You need maximum throughput per GPU and have the cluster to back it
• You want to study production inference optimizations for MoE models

Use vLLM or SGLang if:

• You run consumer or prosumer GPUs (RTX 5090, DGX Spark, Mac Studio)
• You serve multiple models
• You need an API server, KV cache, or load balancing
• You want broad hardware and model support

Use Ollama or llama.cpp if:

• You run a single workstation or laptop
• You want the simplest possible setup
• You serve V4.1-Flash through community GGUF builds

​

FAQ