> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run DeepSeek V4.1 Flash on 2 DGX Sparks: community benchmarks

> Community benchmarks, configuration, and gotchas for running DeepSeek V4.1 Flash (552B MoE) on 2 NVIDIA DGX Sparks with tensor parallelism over RoCE. Includes vLLM and SGLang recipes.

You can run `deepseek-ai/DeepSeek-V4.1-Flash` — a 552B-parameter mixture-of-experts model with native vision, 1M-token context, and MIT licensing — on 2 NVIDIA DGX Sparks connected over a ConnectX-7 cluster link. Multiple independent community teams have published reproducible benchmarks as of October 2026. The model does not fit at full FP8 precision on 256 GB of aggregate unified memory. It fits when you use a mixed-precision serving format that keeps dense layers at FP8 and compresses the expert weights to 4-bit.

This page collects the published numbers, the serving configurations, and the gotchas that every team hit independently. It is a companion to the [V4 Flash 0731 hardware guide](/blog/deepseek-v4-flash-local-hardware.md) for the superseded model.

## What changed from V4 Flash 0731

V4.1-Flash is a fundamentally larger model than the July 31 release it replaces.

| Property | V4-Flash-0731 | V4.1-Flash |
| - | - | - |
| Total parameters | \~284B (+ DSpark head → \~304B) | \~552B backbone + \~196B Engram → \~763B |
| Active per token | 13B | 8B input / 16B output (asymmetric) |
| Weight floor (downloadable) | \~162–167 GB (GGUF) | \~400 GiB (NVFP4 + MXFP4 Engram) |
| KV cache per token | Larger | 890 bytes (3.9× smaller) |
| Context window | 1M | 1M |
| Vision | Text only | Native text + image |
| License | MIT | MIT |

The weight floor more than doubled. The 2 Sparks that comfortably ran V4-Flash-0731 at 162 GB now need a mixed-precision strategy to hold V4.1-Flash at \~400 GiB.

## The memory math for 2 DGX Sparks

Each DGX Spark has 128 GB of LPDDR5x unified memory at 273 GB/s. Two Sparks connected over a ConnectX-7 cluster link give you 256 GB of aggregate memory and \~200 Gb/s of RDMA bandwidth (measured \~183–185 Gb/s in practice).

The NVIDIA [NVFP4 variant](https://huggingface.co/nvidia/DeepSeek-V4.1-Flash-NVFP4) stores expert weights at NVFP4 (W4A4, group size 16), Engram lookup tables at MXFP4 (E2M1), and preserves source precision for attention, shared experts, vision, and MTP/DSpark layers. The current revision (October 2026) totals approximately **400 GiB** on disk. That does not fit in 256 GB.

The community path that works uses a **mixed-precision layout** where the serving framework further compresses expert weights while keeping the dense backbone at higher precision. The result is approximately **79–105 GiB per node** for model weights, leaving 23–49 GB per node for KV cache, runtime buffers, and the operating system.

| Component | Per node (2-node TP=2) |
| - | - |
| Model weights | \~79–105 GiB |
| KV cache | \~15 GiB |
| System and runtime overhead | \~8–12 GB |
| Available headroom | \~10–40 GB |

> [!TIP]
>
> The NVFP4 accuracy retention is strong. NVIDIA published benchmark comparisons showing less than 1% deviation from the MXFP4 source across GPQA Diamond (91.0 → 90.8), SciCode (54.4 → 54.0), and Terminal-Bench 2.1 (81.6 → 81.9). The quality loss from mixed-precision serving is negligible for production work.

## Community benchmarks

Multiple teams have published independent benchmarks. The numbers vary by serving engine, quantization strategy, and concurrency level. All measurements below are decode throughput in tokens per second.

### Single-stream decode (1 concurrent request)

| Source | Engine | Precision | tok/s | Context |
| - | - | - | - | - |
| [r0b0tlab](https://github.com/r0b0tlab/deepseek-v4-flash-nvfp4-gb10-benchmark) | vLLM | FP8 + MXFP4 experts | 38.4 | 65K |
| [flowtivity.ai](https://flowtivity.ai/blog/deepseek-v4-flash-1m-context-dual-dgx-spark/) | vLLM | FP8 dense + MXFP4 experts | 41.1 | 1M |
| [tonyd2wild](https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark) | TensorFold | EXL3 (3.5 bpw) | 73.1 | 1M |
| [Zhihu (Chinese)](https://zhuanlan.zhihu.com/p/2067961897207456860) | vLLM | Mixed | 79.0 | Long output |

### Multi-user aggregate throughput

| Concurrency | tok/s (aggregate) | Source |
| - | - | - |
| 2 users | 104–110 | Level1Techs |
| 4 users | 135–169 | Level1Techs / tonyd2wild |
| 6 users | 219–256 | Level1Techs / NVIDIA forums |
| 16 users | 135–270 | r0b0tlab / tonyd2wild |

### Prefill speed

| Context length | Prefill tok/s | Source |
| - | - | - |
| 8K | \~1,620 | r0b0tlab |
| 32K | \~2,590–2,830 | r0b0tlab |
| 64K | \~4,059–5,000 | r0b0tlab |
| 93K | \~2,215 | tonyd2wild |
| 128K | \~4,900–5,925 | r0b0tlab |

### Time to first token

| Context | TTFT | Source |
| - | - | - |
| 64K | 31.3s | flowtivity.ai |
| 256K | 47.0s | flowtivity.ai |
| 512K | 334.4s | flowtivity.ai |

> [!NOTE]
>
> TTFT at 512K and above is dominated by the prefill pass over the full context. The 2-Spark interconnect at \~185 Gb/s effective throughput is the bottleneck for long prompts. If your workload is latency-sensitive at long context, consider 4 Sparks at TP=4, where prefill scales to \~87 tok/s decode and TTFT drops proportionally.

## Configuration

### vLLM (recommended for 2-Spark TP=2)

vLLM v0.31.0 or later is required. The community consensus configuration uses distributed tensor parallelism across 2 nodes with RoCE networking.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8_ds_mla \
  --block-size 256 \
  --max-model-len 524288 \
  --max-num-seqs 6 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.82 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --distributed-executor-backend mp \
  --nnodes 2
```

Key flags explained:

* `--kv-cache-dtype fp8_ds_mla` — use FP8 for the KV cache, not `nvfp4_ds_mla` (unsupported in stock vLLM)
* `--max-model-len 524288` — 512K is the stable sweet spot; 1M works but costs \~12% decode speed
* `--max-num-seqs 6` — caps concurrent requests; raise to 6 for maximum aggregate throughput
* `--gpu-memory-utilization 0.82` — leaves room for KV cache growth; do not exceed 0.85 on unified memory

### SGLang (recommended for 3–4 Spark TP=3/4)

SGLang with the `dsv4.1` branch supports tensor parallelism of 3 or 4 with NVFP4 quantization.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m sglang.launch_server \
  --model-path nvidia/DeepSeek-V4.1-Flash-NVFP4 \
  --tp 4 \
  --context-length 1048576 \
  --chunked-prefill-size 4096 \
  --max-running-requests 16 \
  --moe-runner-backend flashinfer_cutlass \
  --fp4-gemm-backend flashinfer_cutlass
```

### Networking setup

The ConnectX-7 cluster link uses RoCE v2 (RDMA over Converged Ethernet). Critical environment variables:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export NCCL_IB_HCA=rocep1s0f1
export NCCL_BUFFSIZE=139000000
```

Use GID index 3 for dual-rail configuration. Community reports show +5% decode and +15% long-prompt prefill with dual-rail enabled. The management network runs on `192.168.1.0/24`; model data traffic uses separate subnets (`10.200.0.0/24` through `10.200.5.0/24` for TP=4 ring topology).

## Context length on 2 Sparks

| Target context | Fits? | Decode speed impact | Max concurrent |
| - | - | - | - |
| 64K | Yes | Baseline | 6+ |
| 128K | Yes | \~5% reduction | 4–6 |
| 256K | Yes | \~8% reduction | 3–4 |
| 512K | Yes | \~10% reduction | 2–3 |
| 1M | Yes | \~12% reduction | 1–2 |

The 1M context was verified by multiple teams using needle-in-a-haystack tests. The KV cache at FP8 uses 890 bytes per token, so 1M tokens consumes approximately 0.93 GB of cache memory — modest compared to the weight footprint. The real constraint at 1M is not memory but prefill latency: expect 5+ minutes for the first token at 512K–1M context.

## Critical gotchas

These are the issues that every community team hit independently. Save yourself the debugging time.

### Kernel regression breaks NCCL RoCE

The `7.0.0-1019-nvidia` kernel causes `ibv_reg_mr_iova2 failed with error Cannot allocate memory`. The root cause is that `CmaTotal` drops to 0 kB. The working kernel is `6.17.0-1032-nvidia`.

Pin the kernel and hold the package:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
sudo apt-mark hold linux-nvidia-hwe-24.04
```

The performance impact of the broken kernel is severe: prefill drops from \~2,600 tok/s to \~1,430 tok/s (−47% TTFT).

### PyTorch memory allocator causes NaN logits

Set this environment variable before launching vLLM:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
```

The default `expandable_segments:True` causes NaN logits after approximately 64 generated tokens. This is a PyTorch bug on unified memory (SM121), not a vLLM issue.

### Prefix caching silent failures

Parallel long-context requests can fail silently due to block eviction in the hybrid MLA/SWA attention structure. Set the retention interval:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=4096
```

### DSpark speculative decoding

If you enable DSpark for coding workloads, ensure `DS4_SERVER_COALESCE_MAX` is at least 2. Setting it to 1 silently disables speculation. Use the 0731-specific drafter file — older drafters fail with a missing `dspark.main_proj.weight` tensor error.

DSpark helps coding throughput but may slow non-coding tasks. Test with your actual workload before enabling it in production.

### Build requirements for llama.cpp

If you build llama.cpp from source for GB10 (SM121), you must set:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
cmake -DCMAKE_CUDA_ARCHITECTURES=121 ...
```

Without this, the build silently falls back to a generic CUDA target and performs significantly worse.

## Comparison to other hardware

### 2 DGX Sparks vs Mac Studio M5 Ultra 256 GB

| Metric | 2× DGX Spark | Mac Studio M5 Ultra 256 GB |
| - | - | - |
| Memory | 256 GB (128/node) | 256 GB unified |
| Bandwidth | 273 GB/s per node | 1.2 TB/s |
| V4.1 Flash fits? | Yes (mixed precision) | No (smallest GGUF is 233 GB, leaves no KV headroom) |
| Decode c1 | 38–73 tok/s | N/A |
| Prefill 32K | \~17s | \~50s |
| Setup complexity | High (RoCE, NCCL, kernel pinning) | Low (plug and play) |
| All-in price | \~\$6,000–7,000 | \~\$8,500+ |

V4.1-Flash does not fit on a single 256 GB Mac Studio at any usable quantization. The 512 GB Mac Studio unlocks it at Q2\_K–Q4, but at 2–3× the price of 2 Sparks.

### 2 DGX Sparks vs 4 DGX Sparks

| Metric | 2 Sparks (TP=2) | 4 Sparks (TP=4) |
| - | - | - |
| Decode c1 | 38–73 tok/s | 87–125 tok/s |
| Decode c16 | 135–270 tok/s | 270–343 tok/s |
| Context | 1M (tight) | 1M (comfortable) |
| Memory per node | 79–105 GiB | \~72 GiB |
| KV headroom | 10–40 GB/node | \~24 GB/node |
| Price | \~\$6,000–7,000 | \~\$12,000–14,000 |

4 Sparks roughly doubles single-stream decode and provides comfortable 1M context. 2 Sparks is the minimum viable configuration; 4 Sparks is the recommended production configuration.

### 2 DGX Sparks vs 2× RTX PRO 6000

| Metric | 2× DGX Spark | 2× RTX PRO 6000 |
| - | - | - |
| Memory | 256 GB unified | 192 GB GDDR7 |
| Bandwidth | 273 GB/s | 1,792 GB/s |
| V4.1 Flash fits? | Yes (mixed precision) | Yes (FP4+FP8) |
| Decode c1 | 38–73 tok/s | 106–210 tok/s |
| Aggregate c10 | \~200 tok/s | \~700 tok/s |
| Power draw | \~400W | \~865W |
| Price | \~\$6,000–7,000 | \~\$15,000+ (workstation) |

RTX PRO 6000 is 6.6× faster on memory bandwidth and delivers 2–3× the throughput. DGX Spark wins on price and power efficiency. For production serving with real-time latency requirements, RTX PRO 6000 is the better choice. For always-on personal or small-team use, DGX Spark is more practical.

## Expose through Tokios

Once your 2-Spark cluster is healthy, expose it through a single authenticated endpoint.

**1. Keep the model server on the head node private**

Confirm the vLLM or SGLang process on the head Spark responds on its loopback `/v1` endpoint, such as `http://127.0.0.1:8000/v1`. Worker-to-worker traffic stays on the cluster network.

**2. Pair 1 connector on the head node**

In the [Tokios dashboard](https://tokios.com/console), open **Setup** and select **Start pairing**. Install and run `tokios-connector` on the head Spark. The connector dials out over HTTPS — it does not require an inbound port.

**3. Configure the head-node upstream**

Set the connector's `BaseUrl` to the head node's local `/v1` endpoint. See [Connector config](/reference/connector-config.md) for the complete format.

**4. Register the public deployment**

In **Models**, register a deployment such as `deepseek-v41-flash`. Map it to the upstream model id your head-node server exposes. Clients send the deployment name, not the Hugging Face id.

**5. Create a scoped API key**

Create an `sk-tok-…` key in **Keys**, optionally scoped to `deepseek-v41-flash`. Clients then call `https://api.tokios.com/v1`.

**Verify through Tokios:**

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v41-flash",
    "messages": [{"role": "user", "content": "Summarize this deployment health check."}]
  }'
```

## Validate in this order

1. Pin the kernel to `6.17.0-1032-nvidia` and verify `CmaTotal` is non-zero.
2. Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False`.
3. Complete NVIDIA's 2-Spark networking setup (ConnectX-7, RoCE, subnets).
4. Load the model at 32K context with 1 concurrent request.
5. Send a short chat request, a long prompt, and a tool-call request.
6. Record memory consumption, first-token latency, and generation rate.
7. Raise context toward 128K, then 512K, then 1M — repeating the same workload at each step.
8. Only then test multi-user concurrency.

This order separates kernel and networking failures from memory pressure, KV cache exhaustion, and tool-format issues.

> **[V4 Flash 0731 hardware](/blog/deepseek-v4-flash-local-hardware.md)**
>
> The previous generation guide — 284B model on 2 DGX Sparks, 2 RTX PRO 6000s, or 2 Strix Halo systems.

> **[DGX Spark model compatibility](/guides/dgx-spark-model-compatibility.md)**
>
> What fits on 1 Spark (128 GB) — dense up to 70B Q4, MoE up to 120B NVFP4.

> **[One vs two DGX Sparks](/guides/one-vs-two-dgx-sparks.md)**
>
> When to scale to a second Spark for distributed inference.

> **[DeepSeek family](/models/deepseek.md)**
>
> V4.1-Flash model page with identifiers, capabilities, and the full DeepSeek lineup.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.