> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# DeepInfer: DeepSeek's datacenter inference engine

> DeepInfer is DeepSeek's open-source production inference framework for V4.1-Flash on SM100 datacenter GPUs. Apache 2.0 licensed, but not a vLLM replacement — SM100 only, single model, no API server.

DeepInfer is DeepSeek's in-house inference framework, open-sourced October 2026 under the Apache 2.0 license. It powers the production serving infrastructure behind `chat.deepseek.com`, processing 155 trillion tokens per day. It is **not** a general-purpose serving engine — it is a specialized, single-model optimizer for DeepSeek V4.1-Flash on NVIDIA SM100 datacenter GPUs.

> [!WARNING]
>
> **DeepInfer does not run on consumer hardware.** It requires NVIDIA SM100 architecture (B100, B200, GB200 NVL72) or Huawei Ascend 950 NPUs. It does **not** support SM90 (H100/H200), SM120/SM121 (RTX 5090, DGX Spark GB10), or any consumer/prosumer GPU. If you run a workstation or homelab, use [vLLM](https://docs.tokios.com/guides/vllm-remote-access.md) or [SGLang](https://docs.tokios.com/guides/sglang-remote-access.md) instead.

> [!NOTE]
>
> Last verified: 2026-10-24

## What DeepInfer is

DeepInfer is an offline inference framework — it ships no API server, no KV cache persistence, no prefill/decode disaggregation, and no load balancer. It is the compute kernel layer that sits beneath a serving stack, not a replacement for vLLM or SGLang.

| Property | Value |
| - | - |
| GitHub | [`deepseek-ai/DeepInfer`](https://github.com/deepseek-ai/DeepInfer) |
| License | Apache 2.0 |
| Supported models | **DeepSeek V4.1-Flash only** |
| Supported hardware | NVIDIA SM100 (B100, B200, GB200) or Huawei Ascend 950 |
| Minimum config | 4 SM100 GPUs (prefill), 64 SM100 GPUs (decode) |
| API server | None — offline inference only |
| KV cache persistence | No |
| Load balancing | No |

## Hardware requirements

DeepInfer's SM100 requirement is the most important thing to understand before evaluating it.

| Compute capability | Architecture | GPUs | DeepInfer support |
| - | - | - | - |
| SM100 (10.0) | Blackwell datacenter | B100, B200, GB200 NVL72 | **Yes** |
| SM103 (10.3) | Blackwell datacenter | B300, GB300 | **Yes** |
| SM90 (9.0) | Hopper | H100, H200 | No |
| SM120 (12.0) | Blackwell consumer | RTX 5090, RTX 5080 | No |
| SM121 (12.1) | Blackwell desktop | DGX Spark GB10 | No |

The SM100/SM120 split is the same architecture family (Blackwell) but different silicon. Consumer Blackwell (SM120/121) uses different tensor core instructions that DeepInfer does not target.

**System requirements:**

* Linux (Ubuntu 22.04+ recommended)
* GCC >= 13.0
* Python 3.12
* CUDA 13 (NVIDIA) or CANN 9.2 (Huawei Ascend)

## Performance

DeepInfer's benchmarks come from a 4-GPU prefill / 64-GPU decode cluster running V4.1-Flash:

| Metric | Throughput | Notes |
| - | - | - |
| Prefill | \~60,000–89,000 tok/s/GPU | Input processing |
| Decode | \~2,600–26,000 tok/s/GPU | Scales with request count |
| Per-user decode | \~325 tok/s | At 8 requests per data-parallel group |

These numbers require the full 64-GPU decode cluster. A 4-GPU setup delivers substantially less.

## Key techniques

DeepInfer combines several inference optimizations:

* **FlashMLA** — DeepSeek's custom MLA (Multi-head Latent Attention) kernel for the compressed KV cache
* **DeepGEMM** — Custom GEMM kernels for MoE expert computation
* **Attention DP + MoE EP** — Data parallelism for attention, expert parallelism for MoE layers
* **DSpark verification** — Confidence-scheduled speculative verification using the attached DSpark head
* **SWA bounded replay** — Sliding-window attention with bounded replay buffer
* **Asynchronous scheduling** — Overlapping compute and memory operations
* **Batch-invariant inference** — Deterministic sampling regardless of batch size
* **Structured outputs** — Regex and JSON schema constrained generation

## DeepInfer vs vLLM vs SGLang

| Feature | DeepInfer | vLLM | SGLang |
| - | - | - | - |
| Model support | V4.1-Flash only | 200+ models | 100+ models |
| Hardware | SM100 / Ascend 950 | Broad NVIDIA, AMD, Apple | Broad NVIDIA, AMD |
| API server | None | OpenAI-compatible | OpenAI-compatible |
| KV cache persistence | No | Yes | Yes (RadixAttention) |
| Prefill/decode disaggregation | No | Yes | Yes |
| Load balancing | No | External | External |
| Continuous batching | Yes | Yes | Yes |
| Consumer GPU support | No | Yes | Yes |

DeepInfer is built on vLLM internals and cites SGLang and TensorRT-LLM as inspiration. For general-purpose serving, vLLM or SGLang are the practical choice. DeepInfer is relevant only if you run V4.1-Flash on SM100 or Ascend 950 at datacenter scale.

## Should you use DeepInfer?

**Use DeepInfer if:**

* You operate 64+ SM100 GPUs (B200, GB200) or Ascend 950 NPUs
* You serve DeepSeek V4.1-Flash as your primary model
* You need maximum throughput per GPU and have the cluster to back it
* You want to study production inference optimizations for MoE models

**Use vLLM or SGLang if:**

* You run consumer or prosumer GPUs (RTX 5090, DGX Spark, Mac Studio)
* You serve multiple models
* You need an API server, KV cache, or load balancing
* You want broad hardware and model support

**Use Ollama or llama.cpp if:**

* You run a single workstation or laptop
* You want the simplest possible setup
* You serve V4.1-Flash through community GGUF builds

## FAQ

> **Can I run DeepInfer on RTX 5090?**
>
> No. DeepInfer requires SM100 architecture (B100, B200, GB200), which is datacenter-only. RTX 5090 uses SM120 (consumer Blackwell), a different silicon implementation. Use vLLM or SGLang on RTX 5090 instead.

> **Can I run DeepInfer on DGX Spark?**
>
> No. DGX Spark uses the GB10 chip, which is SM121 (desktop Blackwell). DeepInfer targets SM100 (datacenter Blackwell). Use vLLM or SGLang on DGX Spark — see [V4.1 Flash on 2 DGX Sparks](https://docs.tokios.com/blog/deepseek-v41-flash-dgx-spark.md).

> **Does DeepInfer replace vLLM?**
>
> No. DeepInfer is an offline inference engine for one model on one hardware architecture. vLLM is a general-purpose serving framework with an API server, KV cache, and support for 200+ models. They operate at different layers of the stack.

> **What models does DeepInfer support?**
>
> DeepSeek V4.1-Flash only. No other models are supported.

> **Is DeepInfer open source?**
>
> Yes, Apache 2.0 licensed. The source is at [`deepseek-ai/DeepInfer`](https://github.com/deepseek-ai/DeepInfer).

> **[DeepSeek V4.1-Flash](https://docs.tokios.com/models/deepseek.md)**
>
> The model DeepInfer serves — specs, hardware requirements, and local setup.

> **[V4.1 Flash on 2 DGX Sparks](https://docs.tokios.com/blog/deepseek-v41-flash-dgx-spark.md)**
>
> Consumer-friendly path to V4.1-Flash using vLLM and SGLang.

> **[vLLM remote access](https://docs.tokios.com/guides/vllm-remote-access.md)**
>
> General-purpose serving with an OpenAI-compatible API.

> **[SGLang remote access](https://docs.tokios.com/guides/sglang-remote-access.md)**
>
> High-throughput serving with RadixAttention.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.