Install and configure vLLM on NVIDIA DGX Spark

vLLM with NVFP4 quantization is the best-performing general-purpose serving path on DGX Spark. It gives you an OpenAI-compatible /v1 endpoint, native SM121 support, and the throughput numbers the community reports — 47-50 tok/s on Qwen3.5-35B-A3B, 15-20 tok/s on 70B dense, and ~30 tok/s on 120B MoE. This page walks you through installation, first launch, and the settings that matter.

Alternative engines. Some community members report better results with SGLang on specific model configurations, and the Atlas engine claims 2.8x over stock vLLM for Qwen3.6-35B-A3B. This page covers stock vLLM because it has the broadest model support and the most documented configuration surface.

Prerequisites

Before you start, confirm your DGX Spark meets these requirements:

OS: DGX OS (Ubuntu-based, Linux aarch64)
CUDA: 13.0 or later — CUDA 12 libraries fail on SM121
Python: 3.10 or later (for pip install) or Docker with ARM64 support
Memory: 128 GB unified (default on all DGX Spark / GB10 units)

Verify your CUDA version:

nvcc --version

Verify the GPU is visible:

nvidia-smi

You should see the GB10 Blackwell GPU with 128 GB reported.

DGX Spark requires CUDA 13.0 exclusively. The SM121 architecture does not support CUDA 12 libraries. If nvcc --version shows CUDA 12.x, update your DGX OS packages before proceeding.

Install vLLM

Install vLLM via pip

The simplest path on DGX OS is a direct pip install. vLLM ships ARM64 wheels that work on the Grace CPU.

pip install vllm

If you need the nightly build for the latest NVFP4 or SM121 fixes:

pip install --pre vllm --extra-index-url https://wheels.vllm.ai/nightly

Or pull the Docker image

If you prefer containerized deployment, pull the ARM64-compatible image:

docker pull vllm/vllm-openai:latest

Run with GPU access:

docker run --gpus all -it --rm -p 8000:8000 vllm/vllm-openai:latest

Verify the installation

Confirm vLLM starts and detects the GPU:

python -c "import vllm; print(vllm.__version__)"

If this prints a version number without CUDA errors, you are ready to serve.

Launch your first model

Start with a 35B MoE model — it is fast, well-tested, and leaves plenty of memory headroom.

vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768

This launches an OpenAI-compatible server on http://127.0.0.1:8000. Test it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Qwen3.5-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 64
  }'

You should see a response within a few seconds, with time-to-first-token around 0.12 seconds.

Optimal settings by model size

The 3 settings that matter most on DGX Spark are --gpu-memory-utilization, --max-model-len, and --quantization. Here are tested starting points:

Model size	`gpu_memory_utilization`	`max_model_len`	`quantization`	Expected tok/s
35B MoE	0.90	32768	`nvfp4`	47-50
70B dense	0.85	8192	`nvfp4`	15-20
120B MoE	0.85	8192	`nvfp4`	25-30

35B MoE (Qwen3.5-35B-A3B, Qwen3.6-35B-A3B)

vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768

35B MoE models use only ~18 GB for weights, leaving substantial room for KV cache. You can push max-model-len higher if you need longer context.

70B dense

vllm serve <your-70b-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192

70B dense models use ~35 GB for weights at NVFP4. Keep max-model-len at 8192 to leave room for KV cache across concurrent requests. Community reports show multi-user concurrency is limited to 5-10 users at this size.

120B MoE (Qwen3.5-122B-A10B, Nemotron-3-Super-120B)

vllm serve <your-120b-moe-model> \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192

120B MoE models use ~60 GB for weights. The 8192 context limit keeps KV cache manageable. Expect ~30 tok/s for single-user inference.

For detailed memory tuning — KV cache sizing, batch size trade-offs, and --kv-cache-dtype fp8 — see DGX Spark memory tuning.

NVFP4 configuration recipes

NVFP4 is the recommended quantization for DGX Spark. It runs natively on SM121 and gives a 75% size reduction with low quality impact.

Use a pre-quantized model

NVIDIA publishes NVFP4 checkpoints on Hugging Face. Use the nvidia/ prefix:

vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 --quantization nvfp4

Online quantization at load time

If no pre-quantized checkpoint exists, vLLM can quantize on the fly from a BF16 model:

vllm serve <model-path> --quantization nvfp4

This takes longer to load (the model must be quantized in memory first) but works for any model architecture vLLM supports.

SGLang alternative

Some community members recommend SGLang over stock vLLM for specific DGX Spark workloads. SGLang runs on the same CUDA 13.0 stack and uses the same model formats.

python -m sglang.launch_server --model-path <model> --port 8000

Consider SGLang if vLLM’s default configuration does not meet your throughput requirements for a specific model. The configuration surface is different — check SGLang’s own documentation for DGX Spark-specific flags.

ARM64 and SM121 gotchas

DGX Spark’s Grace CPU (ARM64) and Blackwell GPU (SM121) have a few platform-specific issues worth knowing before you hit them.

Issue	Symptom	Fix
CUDA 12 libraries	Import errors, kernel launch failures	Use CUDA 13.0 exclusively; update DGX OS
NIM containers fail	Container exits or errors on ARM64	Use vLLM directly instead of NIM containers
OOM without clean error	System becomes unresponsive instead of throwing CUDA OOM	Cap `gpu_memory_utilization` at 0.85-0.90; see memory tuning
Gemma-4 `gpu_memory_utilization`	Parameter ignored during startup profiling, system hangs	Use NVFP4 quantization explicitly; track vLLM #46307

DGX Spark becomes unresponsive instead of throwing a CUDA OOM error. On unified memory systems, when the model plus KV cache exceeds available memory, the system hangs rather than failing gracefully. Always start with conservative gpu_memory_utilization values (0.85 for large models) and increase only after confirming stability.

Benchmark your setup

After launching, verify throughput with a quick benchmark:

python -m vllm.entrypoints.openai.api_server \
  --model nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --quantization nvfp4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 &

# Wait for the server to start, then:
python -m vllm.benchmarks.benchmark_serving \
  --model nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --num-prompts 10 \
  --request-rate 1

Compare your results against the expected tok/s in the settings table above. If throughput is significantly lower, check that CUDA 13.0 is active and that no other processes are consuming GPU memory. Monitor memory usage while serving:

watch -n 1 nvidia-smi

Expose your endpoint with Tokios

vLLM on the Spark gives you a local /v1 endpoint. To reach it from your laptop, coding agents, or teammates without SSH tunnels or open ports, run the Tokios connector alongside vLLM.

Start vLLM on the Spark

Launch your model as shown above. Confirm it answers on http://127.0.0.1:8000/v1.

Install and pair the connector

In the Tokios dashboard, open Setup and select Start pairing to get a claim code. On the Spark:

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

On the Models tab, map your vLLM model id to a deployment name. Clients use the deployment name, not the upstream model id.

See the full DGX Spark use-case guide for client configuration examples.

Model compatibility

Check which models fit in 128GB and their expected throughput.

Memory tuning

KV cache sizing, FP8 KV cache, and tuning parameters for production workloads.

vLLM remote access

General vLLM remote access patterns beyond DGX Spark.

One vs two Sparks

Scale to 405B models with a second Spark and tensor parallelism.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Install and configure vLLM on NVIDIA DGX Spark

Prerequisites

Install vLLM

Launch your first model

Optimal settings by model size

35B MoE (Qwen3.5-35B-A3B, Qwen3.6-35B-A3B)

70B dense

120B MoE (Qwen3.5-122B-A10B, Nemotron-3-Super-120B)

NVFP4 configuration recipes

Use a pre-quantized model

Online quantization at load time

SGLang alternative

ARM64 and SM121 gotchas

Benchmark your setup

Expose your endpoint with Tokios

Model compatibility

Memory tuning

vLLM remote access

One vs two Sparks

​Prerequisites

​Install vLLM

​Launch your first model

​Optimal settings by model size

​35B MoE (Qwen3.5-35B-A3B, Qwen3.6-35B-A3B)

​70B dense

​120B MoE (Qwen3.5-122B-A10B, Nemotron-3-Super-120B)

​NVFP4 configuration recipes

​Use a pre-quantized model

​Online quantization at load time

​SGLang alternative

​ARM64 and SM121 gotchas

​Benchmark your setup

​Expose your endpoint with Tokios

Model compatibility

Memory tuning

vLLM remote access

One vs two Sparks

Prerequisites

Install vLLM

Launch your first model

Optimal settings by model size

35B MoE (Qwen3.5-35B-A3B, Qwen3.6-35B-A3B)

70B dense

120B MoE (Qwen3.5-122B-A10B, Nemotron-3-Super-120B)

NVFP4 configuration recipes

Use a pre-quantized model

Online quantization at load time

SGLang alternative

ARM64 and SM121 gotchas

Benchmark your setup

Expose your endpoint with Tokios