Run DeepSeek V4.1 Flash on 2 DGX Sparks: community benchmarks

You can run deepseek-ai/DeepSeek-V4.1-Flash — a 552B-parameter mixture-of-experts model with native vision, 1M-token context, and MIT licensing — on 2 NVIDIA DGX Sparks connected over a ConnectX-7 cluster link. Multiple independent community teams have published reproducible benchmarks as of October 2026. The model does not fit at full FP8 precision on 256 GB of aggregate unified memory. It fits when you use a mixed-precision serving format that keeps dense layers at FP8 and compresses the expert weights to 4-bit. This page collects the published numbers, the serving configurations, and the gotchas that every team hit independently. It is a companion to the V4 Flash 0731 hardware guide for the superseded model.

What changed from V4 Flash 0731

V4.1-Flash is a fundamentally larger model than the July 31 release it replaces.

Property	V4-Flash-0731	V4.1-Flash
Total parameters	~284B (+ DSpark head → ~304B)	~552B backbone + ~196B Engram → ~763B
Active per token	13B	8B input / 16B output (asymmetric)
Weight floor (downloadable)	~162–167 GB (GGUF)	~400 GiB (NVFP4 + MXFP4 Engram)
KV cache per token	Larger	890 bytes (3.9× smaller)
Context window	1M	1M
Vision	Text only	Native text + image
License	MIT	MIT

The weight floor more than doubled. The 2 Sparks that comfortably ran V4-Flash-0731 at 162 GB now need a mixed-precision strategy to hold V4.1-Flash at ~400 GiB.

The memory math for 2 DGX Sparks

Each DGX Spark has 128 GB of LPDDR5x unified memory at 273 GB/s. Two Sparks connected over a ConnectX-7 cluster link give you 256 GB of aggregate memory and ~200 Gb/s of RDMA bandwidth (measured ~183–185 Gb/s in practice). The NVIDIA NVFP4 variant stores expert weights at NVFP4 (W4A4, group size 16), Engram lookup tables at MXFP4 (E2M1), and preserves source precision for attention, shared experts, vision, and MTP/DSpark layers. The current revision (October 2026) totals approximately 400 GiB on disk. That does not fit in 256 GB. The community path that works uses a mixed-precision layout where the serving framework further compresses expert weights while keeping the dense backbone at higher precision. The result is approximately 79–105 GiB per node for model weights, leaving 23–49 GB per node for KV cache, runtime buffers, and the operating system.

Component	Per node (2-node TP=2)
Model weights	~79–105 GiB
KV cache	~15 GiB
System and runtime overhead	~8–12 GB
Available headroom	~10–40 GB

The NVFP4 accuracy retention is strong. NVIDIA published benchmark comparisons showing less than 1% deviation from the MXFP4 source across GPQA Diamond (91.0 → 90.8), SciCode (54.4 → 54.0), and Terminal-Bench 2.1 (81.6 → 81.9). The quality loss from mixed-precision serving is negligible for production work.

Community benchmarks

Multiple teams have published independent benchmarks. The numbers vary by serving engine, quantization strategy, and concurrency level. All measurements below are decode throughput in tokens per second.

Single-stream decode (1 concurrent request)

Source	Engine	Precision	tok/s	Context
r0b0tlab	vLLM	FP8 + MXFP4 experts	38.4	65K
flowtivity.ai	vLLM	FP8 dense + MXFP4 experts	41.1	1M
tonyd2wild	TensorFold	EXL3 (3.5 bpw)	73.1	1M
Zhihu (Chinese)	vLLM	Mixed	79.0	Long output

Multi-user aggregate throughput

Concurrency	tok/s (aggregate)	Source
2 users	104–110	Level1Techs
4 users	135–169	Level1Techs / tonyd2wild
6 users	219–256	Level1Techs / NVIDIA forums
16 users	135–270	r0b0tlab / tonyd2wild

Prefill speed

Context length	Prefill tok/s	Source
8K	~1,620	r0b0tlab
32K	~2,590–2,830	r0b0tlab
64K	~4,059–5,000	r0b0tlab
93K	~2,215	tonyd2wild
128K	~4,900–5,925	r0b0tlab

Time to first token

Context	TTFT	Source
64K	31.3s	flowtivity.ai
256K	47.0s	flowtivity.ai
512K	334.4s	flowtivity.ai

TTFT at 512K and above is dominated by the prefill pass over the full context. The 2-Spark interconnect at ~185 Gb/s effective throughput is the bottleneck for long prompts. If your workload is latency-sensitive at long context, consider 4 Sparks at TP=4, where prefill scales to ~87 tok/s decode and TTFT drops proportionally.

Configuration

vLLM (recommended for 2-Spark TP=2)

vLLM v0.31.0 or later is required. The community consensus configuration uses distributed tensor parallelism across 2 nodes with RoCE networking.

vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8_ds_mla \
  --block-size 256 \
  --max-model-len 524288 \
  --max-num-seqs 6 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.82 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --distributed-executor-backend mp \
  --nnodes 2

Key flags explained:

--kv-cache-dtype fp8_ds_mla — use FP8 for the KV cache, not nvfp4_ds_mla (unsupported in stock vLLM)
--max-model-len 524288 — 512K is the stable sweet spot; 1M works but costs ~12% decode speed
--max-num-seqs 6 — caps concurrent requests; raise to 6 for maximum aggregate throughput
--gpu-memory-utilization 0.82 — leaves room for KV cache growth; do not exceed 0.85 on unified memory

SGLang (recommended for 3–4 Spark TP=3/4)

SGLang with the dsv4.1 branch supports tensor parallelism of 3 or 4 with NVFP4 quantization.

python -m sglang.launch_server \
  --model-path nvidia/DeepSeek-V4.1-Flash-NVFP4 \
  --tp 4 \
  --context-length 1048576 \
  --chunked-prefill-size 4096 \
  --max-running-requests 16 \
  --moe-runner-backend flashinfer_cutlass \
  --fp4-gemm-backend flashinfer_cutlass

Networking setup

The ConnectX-7 cluster link uses RoCE v2 (RDMA over Converged Ethernet). Critical environment variables:

export NCCL_IB_HCA=rocep1s0f1
export NCCL_BUFFSIZE=139000000

Use GID index 3 for dual-rail configuration. Community reports show +5% decode and +15% long-prompt prefill with dual-rail enabled. The management network runs on 192.168.1.0/24; model data traffic uses separate subnets (10.200.0.0/24 through 10.200.5.0/24 for TP=4 ring topology).

Context length on 2 Sparks

Target context	Fits?	Decode speed impact	Max concurrent
64K	Yes	Baseline	6+
128K	Yes	~5% reduction	4–6
256K	Yes	~8% reduction	3–4
512K	Yes	~10% reduction	2–3
1M	Yes	~12% reduction	1–2

The 1M context was verified by multiple teams using needle-in-a-haystack tests. The KV cache at FP8 uses 890 bytes per token, so 1M tokens consumes approximately 0.93 GB of cache memory — modest compared to the weight footprint. The real constraint at 1M is not memory but prefill latency: expect 5+ minutes for the first token at 512K–1M context.

Critical gotchas

These are the issues that every community team hit independently. Save yourself the debugging time.

Kernel regression breaks NCCL RoCE

The 7.0.0-1019-nvidia kernel causes ibv_reg_mr_iova2 failed with error Cannot allocate memory. The root cause is that CmaTotal drops to 0 kB. The working kernel is 6.17.0-1032-nvidia. Pin the kernel and hold the package:

sudo apt-mark hold linux-nvidia-hwe-24.04

The performance impact of the broken kernel is severe: prefill drops from ~2,600 tok/s to ~1,430 tok/s (−47% TTFT).

PyTorch memory allocator causes NaN logits

Set this environment variable before launching vLLM:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False

The default expandable_segments:True causes NaN logits after approximately 64 generated tokens. This is a PyTorch bug on unified memory (SM121), not a vLLM issue.

Prefix caching silent failures

Parallel long-context requests can fail silently due to block eviction in the hybrid MLA/SWA attention structure. Set the retention interval:

export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=4096

DSpark speculative decoding

If you enable DSpark for coding workloads, ensure DS4_SERVER_COALESCE_MAX is at least 2. Setting it to 1 silently disables speculation. Use the 0731-specific drafter file — older drafters fail with a missing dspark.main_proj.weight tensor error. DSpark helps coding throughput but may slow non-coding tasks. Test with your actual workload before enabling it in production.

Build requirements for llama.cpp

If you build llama.cpp from source for GB10 (SM121), you must set:

cmake -DCMAKE_CUDA_ARCHITECTURES=121 ...

Without this, the build silently falls back to a generic CUDA target and performs significantly worse.

Comparison to other hardware

2 DGX Sparks vs Mac Studio M5 Ultra 256 GB

Metric	2× DGX Spark	Mac Studio M5 Ultra 256 GB
Memory	256 GB (128/node)	256 GB unified
Bandwidth	273 GB/s per node	1.2 TB/s
V4.1 Flash fits?	Yes (mixed precision)	No (smallest GGUF is 233 GB, leaves no KV headroom)
Decode c1	38–73 tok/s	N/A
Prefill 32K	~17s	~50s
Setup complexity	High (RoCE, NCCL, kernel pinning)	Low (plug and play)
All-in price	~$6,000–7,000	~$8,500+

V4.1-Flash does not fit on a single 256 GB Mac Studio at any usable quantization. The 512 GB Mac Studio unlocks it at Q2_K–Q4, but at 2–3× the price of 2 Sparks.

2 DGX Sparks vs 4 DGX Sparks

Metric	2 Sparks (TP=2)	4 Sparks (TP=4)
Decode c1	38–73 tok/s	87–125 tok/s
Decode c16	135–270 tok/s	270–343 tok/s
Context	1M (tight)	1M (comfortable)
Memory per node	79–105 GiB	~72 GiB
KV headroom	10–40 GB/node	~24 GB/node
Price	~$6,000–7,000	~$12,000–14,000

4 Sparks roughly doubles single-stream decode and provides comfortable 1M context. 2 Sparks is the minimum viable configuration; 4 Sparks is the recommended production configuration.

2 DGX Sparks vs 2× RTX PRO 6000

Metric	2× DGX Spark	2× RTX PRO 6000
Memory	256 GB unified	192 GB GDDR7
Bandwidth	273 GB/s	1,792 GB/s
V4.1 Flash fits?	Yes (mixed precision)	Yes (FP4+FP8)
Decode c1	38–73 tok/s	106–210 tok/s
Aggregate c10	~200 tok/s	~700 tok/s
Power draw	~400W	~865W
Price	~$6,000–7,000	~$15,000+ (workstation)

RTX PRO 6000 is 6.6× faster on memory bandwidth and delivers 2–3× the throughput. DGX Spark wins on price and power efficiency. For production serving with real-time latency requirements, RTX PRO 6000 is the better choice. For always-on personal or small-team use, DGX Spark is more practical.

Expose through Tokios

Once your 2-Spark cluster is healthy, expose it through a single authenticated endpoint.

Keep the model server on the head node private

Confirm the vLLM or SGLang process on the head Spark responds on its loopback /v1 endpoint, such as http://127.0.0.1:8000/v1. Worker-to-worker traffic stays on the cluster network.

Pair 1 connector on the head node

In the Tokios dashboard, open Setup and select Start pairing. Install and run tokios-connector on the head Spark. The connector dials out over HTTPS — it does not require an inbound port.

Configure the head-node upstream

Set the connector’s BaseUrl to the head node’s local /v1 endpoint. See Connector config for the complete format.

In Models, register a deployment such as deepseek-v41-flash. Map it to the upstream model id your head-node server exposes. Clients send the deployment name, not the Hugging Face id.

Create a scoped API key

Create an sk-tok-… key in Keys, optionally scoped to deepseek-v41-flash. Clients then call https://api.tokios.com/v1.

Verify through Tokios:

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v41-flash",
    "messages": [{"role": "user", "content": "Summarize this deployment health check."}]
  }'

Validate in this order

Pin the kernel to 6.17.0-1032-nvidia and verify CmaTotal is non-zero.
Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False.
Complete NVIDIA’s 2-Spark networking setup (ConnectX-7, RoCE, subnets).
Load the model at 32K context with 1 concurrent request.
Send a short chat request, a long prompt, and a tool-call request.
Record memory consumption, first-token latency, and generation rate.
Raise context toward 128K, then 512K, then 1M — repeating the same workload at each step.
Only then test multi-user concurrency.

This order separates kernel and networking failures from memory pressure, KV cache exhaustion, and tool-format issues.

V4 Flash 0731 hardware

The previous generation guide — 284B model on 2 DGX Sparks, 2 RTX PRO 6000s, or 2 Strix Halo systems.

DGX Spark model compatibility

What fits on 1 Spark (128 GB) — dense up to 70B Q4, MoE up to 120B NVFP4.

One vs two DGX Sparks

When to scale to a second Spark for distributed inference.

DeepSeek family

V4.1-Flash model page with identifiers, capabilities, and the full DeepSeek lineup.

​What changed from V4 Flash 0731

​The memory math for 2 DGX Sparks

​Community benchmarks

​Single-stream decode (1 concurrent request)

​Multi-user aggregate throughput

​Prefill speed

​Time to first token

​Configuration

​vLLM (recommended for 2-Spark TP=2)

​SGLang (recommended for 3–4 Spark TP=3/4)

​Networking setup

​Context length on 2 Sparks

​Critical gotchas

​Kernel regression breaks NCCL RoCE

​PyTorch memory allocator causes NaN logits

​Prefix caching silent failures

​DSpark speculative decoding

​Build requirements for llama.cpp

​Comparison to other hardware

​2 DGX Sparks vs Mac Studio M5 Ultra 256 GB

​2 DGX Sparks vs 4 DGX Sparks

​2 DGX Sparks vs 2× RTX PRO 6000

​Expose through Tokios

​Validate in this order

V4 Flash 0731 hardware

DGX Spark model compatibility

One vs two DGX Sparks

DeepSeek family

What changed from V4 Flash 0731

The memory math for 2 DGX Sparks

Community benchmarks

Single-stream decode (1 concurrent request)

Multi-user aggregate throughput

Prefill speed

Time to first token

Configuration

vLLM (recommended for 2-Spark TP=2)

SGLang (recommended for 3–4 Spark TP=3/4)

Networking setup

Context length on 2 Sparks

Critical gotchas

Kernel regression breaks NCCL RoCE

PyTorch memory allocator causes NaN logits

Prefix caching silent failures

DSpark speculative decoding

Build requirements for llama.cpp

Comparison to other hardware

2 DGX Sparks vs Mac Studio M5 Ultra 256 GB

2 DGX Sparks vs 4 DGX Sparks

2 DGX Sparks vs 2× RTX PRO 6000

Expose through Tokios

Validate in this order