DGX Spark memory tuning and benchmarks

DGX Spark’s 128GB unified memory pool is both its biggest advantage and its biggest tuning challenge. Unlike discrete GPUs where VRAM is fixed and separate from system RAM, the Grace CPU and Blackwell GPU share one pool — and when you run out, the system becomes unresponsive instead of throwing a clean out-of-memory error. This page explains where the memory goes, how to size the KV cache, and which vLLM parameters to tune for your workload.

How unified memory works on GB10

DGX Spark uses NVIDIA’s GB10 chip, which combines a Grace ARM CPU and a Blackwell GPU on one package. They share 128GB of LPDDR5X memory over a 273 GB/s interconnect. There is no separate VRAM. When vLLM allocates “GPU memory,” it is carving a region out of the same 128GB pool that DGX OS, the CUDA runtime, and your applications use. This means:

gpu_memory_utilization controls a share of the total pool, not a dedicated GPU memory budget.
System processes compete for the same bytes. DGX OS, background services, and CUDA context overhead all reduce what the model can use.
There is no OOM killer safety net. When the pool is exhausted, DGX Spark hangs rather than terminating the offending process.

A realistic memory budget looks like this:

Allocation	Typical size
DGX OS and system services	4-6 GB
CUDA runtime and context	2-3 GB
vLLM CUDA graphs	3-5 GB
Available for model + KV cache	~115 GB

KV cache overhead on 128GB

The KV cache stores the key-value tensors from every token in every active sequence. Its size grows with 3 factors: context length, number of concurrent sequences, and the model’s architecture (number of layers and KV heads).

The formula

For a single sequence at a given context length:

KV cache per sequence = 2 × layers × KV_heads × head_dim × context_length × bytes_per_element

The factor of 2 accounts for both keys and values.

Example: 70B dense model at different context lengths

Assuming a typical 70B model with 80 layers, 8 KV heads, head dimension 128:

Context length	KV cache (FP16)	KV cache (FP8)	Notes
4K tokens	~5.2 GB	~2.6 GB	Comfortable for single user
8K tokens	~10.5 GB	~5.2 GB	Recommended limit for multi-user
16K tokens	~21 GB	~10.5 GB	Tight — only 1-2 concurrent users
32K tokens	~42 GB	~21 GB	Single user only on 128GB

Use FP8 KV cache to cut memory in half. Set --kv-cache-dtype fp8 in your vLLM launch command. The quality impact is less than 2% on most benchmarks, and it effectively doubles the number of concurrent sequences or context length you can support.

Example: 35B MoE model

MoE models have the same KV cache math as dense models for their total layer count. A 35B MoE model with ~64 layers at the same KV head configuration:

Context length	KV cache (FP16)	KV cache (FP8)
8K tokens	~4.2 GB	~2.1 GB
32K tokens	~16.8 GB	~8.4 GB
128K tokens	~67.2 GB	~33.6 GB

At 35B MoE with NVFP4 weights (~18 GB), you can fit 128K context with FP8 KV cache and still have ~65 GB of headroom.

Context length vs batch size trade-off

Every token of context in every active sequence consumes KV cache memory. The total KV cache budget is:

Total KV cache = KV per sequence × max_num_seqs × max_model_len / sequence_length

In practice, the trade-off on DGX Spark looks like this:

Priority	You can have	But you sacrifice
Long context (32K+)	1-2 concurrent users	Throughput under load
High concurrency (10+ users)	Short context (4K-8K)	Document-length tasks
Balanced (5 users, 8K context)	Reasonable for both	Neither extreme

For a 70B dense model at NVFP4 (~35 GB weights, ~75 GB remaining for KV cache):

`max_num_seqs`	`max_model_len`	FP16 KV fits?	FP8 KV fits?
5	4096	Yes (~26 GB)	Yes (~13 GB)
5	8192	Tight (~52 GB)	Yes (~26 GB)
10	4096	Tight (~52 GB)	Yes (~26 GB)
10	8192	No (~105 GB)	Tight (~52 GB)

Real-world benchmarks vs NVIDIA claims

NVIDIA’s published figure of 82,739 tok/s for DGX Spark refers to aggregate fine-tuning throughput, not single-user inference. The real numbers for interactive serving are:

Model	Quantization	tok/s (single user)	tok/s (5 concurrent)	Notes
Qwen3.5-35B-A3B	NVFP4 (vLLM)	47-50	Not measured	Sustained; TTFT 0.12s
Qwen3.6-35B-A3B	NVFP4 (Atlas)	~120	Not measured	2.8x stock vLLM; optimized engine
Qwen3.5-122B-A10B	NVFP4 (vLLM)	~30	Not measured	Breaks under 100K concurrent load
70B dense	NVFP4 (vLLM)	15-20	~5-10 users max	Concurrency-limited by KV cache
Qwen3-32B	Q4_K_M (Ollama)	9-44	Not measured	Wide variance by config

The 82,739 tok/s figure measures aggregate throughput across many parallel fine-tuning streams. Single-user interactive inference is a different workload — expect 15-120 tok/s depending on model size and engine, which is still fast enough for real-time chat and coding agent use.

Memory guard: prevent system hangs

The most important tuning parameter on DGX Spark is --gpu-memory-utilization. Set it too high and the system hangs without a recoverable error. Set it conservatively and you lose some KV cache headroom.

Recommended starting values

Model size	`gpu_memory_utilization`	Rationale
≤35B (MoE or dense)	0.90	Plenty of headroom; weights are small
70B dense	0.85	Leave ~15 GB for system and KV cache overflow
120B MoE	0.85	Weights alone use ~60 GB; keep reserve
Any model + long context	0.80	Extra safety margin for large KV caches

Never set gpu_memory_utilization above 0.90 on DGX Spark. Unlike discrete GPUs, unified memory exhaustion makes the entire system unresponsive. You will need a hard power cycle to recover. Start conservative and increase only after confirming stability under your actual workload.

Monitor memory in real time

Open a second terminal and watch memory usage while serving:

watch -n 1 "nvidia-smi --query-gpu=memory.used,memory.free,memory.total --format=csv"

If memory.free drops below 2-3 GB, your system is at risk of hanging. Reduce max_model_len, max_num_seqs, or gpu_memory_utilization.

Tuning parameters reference

These are the vLLM flags that matter most on DGX Spark, in order of impact:

Parameter	What it controls	DGX Spark guidance
`--gpu-memory-utilization`	Fraction of memory vLLM can use	0.80-0.90; never above 0.90
`--max-model-len`	Maximum sequence length	8192 for 70B+; 32768 for 35B MoE
`--max-num-seqs`	Maximum concurrent sequences	5-10 for 70B; higher for 35B MoE
`--kv-cache-dtype`	KV cache precision	`fp8` to halve KV cache memory
`--swap-space`	CPU swap for evicted sequences (GB)	0 on DGX Spark — swapping to shared memory hurts
`--quantization`	Weight quantization	`nvfp4` is recommended for SM121
`--enable-prefix-caching`	Reuse KV cache across prompts	Enable for repeated system prompts

Decision table: common configurations

Scenario	`gpu_memory_utilization`	`max_model_len`	`max_num_seqs`	`kv_cache_dtype`
35B MoE, single user, long context	0.90	32768	1	`auto`
35B MoE, 5 users, 8K context	0.90	8192	5	`fp8`
70B dense, single user	0.85	8192	1	`fp8`
70B dense, 5 users	0.85	4096	5	`fp8`
120B MoE, single user	0.85	8192	1	`fp8`

Tuning recipes

Maximize throughput for a single user

vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-seqs 1 \
  --enable-prefix-caching

Maximize concurrent users for 70B

vllm serve <your-70b-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 4096 \
  --max-num-seqs 10 \
  --kv-cache-dtype fp8

Safest configuration for large models

vllm serve <your-120b-moe-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.80 \
  --max-model-len 8192 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8 \
  --swap-space 0

Next steps

vLLM on DGX Spark

Installation and launch commands for every model size.

Model compatibility

Which models fit in 128GB and their quantization options.

One vs two Sparks

When memory tuning isn’t enough and you need a second Spark.

KV cache concepts

Deeper explanation of KV cache mechanics across model architectures.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

DGX Spark memory tuning and benchmarks

How unified memory works on GB10

KV cache overhead on 128GB

The formula

Example: 70B dense model at different context lengths

Example: 35B MoE model

Context length vs batch size trade-off

Real-world benchmarks vs NVIDIA claims

Memory guard: prevent system hangs

Recommended starting values

Monitor memory in real time

Tuning parameters reference

Decision table: common configurations

Tuning recipes

Maximize throughput for a single user

Maximize concurrent users for 70B

Safest configuration for large models

Next steps

vLLM on DGX Spark

Model compatibility

One vs two Sparks

KV cache concepts

​How unified memory works on GB10

​KV cache overhead on 128GB

​The formula

​Example: 70B dense model at different context lengths

​Example: 35B MoE model

​Context length vs batch size trade-off

​Real-world benchmarks vs NVIDIA claims

​Memory guard: prevent system hangs

​Recommended starting values

​Monitor memory in real time

​Tuning parameters reference

​Decision table: common configurations

​Tuning recipes

​Maximize throughput for a single user

​Maximize concurrent users for 70B

​Safest configuration for large models

​Next steps

vLLM on DGX Spark

Model compatibility

One vs two Sparks

KV cache concepts

How unified memory works on GB10

KV cache overhead on 128GB

The formula

Example: 70B dense model at different context lengths

Example: 35B MoE model

Context length vs batch size trade-off

Real-world benchmarks vs NVIDIA claims

Memory guard: prevent system hangs

Recommended starting values

Monitor memory in real time

Tuning parameters reference

Decision table: common configurations

Tuning recipes

Maximize throughput for a single user

Maximize concurrent users for 70B

Safest configuration for large models

Next steps