> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run Qwen3.6-35B-A3B on DGX Spark

> Step-by-step guide to running Qwen3.6-35B-A3B (35B MoE, 3B active) on NVIDIA DGX Spark with vLLM and Tokios. Achieve 47-120 tok/s with NVFP4 quantization.

Qwen3.6-35B-A3B is the sweet-spot model for DGX Spark. It's a 35B-parameter mixture-of-experts model with only 3B active parameters per token, giving you large-model capability with small-model speed. On a single DGX Spark with NVFP4 quantization, you get 47-50 tok/s sustained throughput — or up to 120 tok/s with the Atlas engine.

This guide walks you through downloading the model, serving it with vLLM, and exposing it through Tokios for team access.

Last verified: 2026-10-24

## What you'll get

* **Model:** Qwen3.6-35B-A3B (35B total, 3B active per token)
* **Hardware:** Single NVIDIA DGX Spark (128 GB unified memory)
* **Throughput:** 47-50 tok/s with vLLM, up to 120 tok/s with Atlas
* **Context:** 32K+ tokens
* **Memory usage:** \~18 GB for weights, leaving 100+ GB for KV cache
* **Use case:** High-speed coding assistant, chat, JSON extraction, agentic workflows

## Prerequisites

* NVIDIA DGX Spark with DGX OS installed
* Internet connection for downloading model weights (\~18 GB)
* Tokios account (free tier works)
* 15 minutes

## Step 1: Install vLLM

DGX Spark ships with CUDA and Python. Install vLLM v0.31.0 or later:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install --upgrade vllm
```

Verify the installation:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm --version
```

## Step 2: Download the model

The NVFP4 quantization is natively supported by the Blackwell GPU's SM121 architecture and gives 75% size reduction with minimal quality loss.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
huggingface-cli download Qwen/Qwen3.6-35B-A3B-NVFP4
```

The download is \~18 GB. On DGX Spark's 273 GB/s memory bandwidth, this takes about 40 seconds.

> [!NOTE]
>
> If you prefer GGUF format for llama.cpp or Ollama, use `Qwen/Qwen3.6-35B-A3B-GGUF` instead. However, NVFP4 is recommended for DGX Spark because it's natively supported by the hardware and works with both vLLM and Atlas.

## Step 3: Serve the model with vLLM

Start the vLLM server with the NVFP4 model:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve Qwen/Qwen3.6-35B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90
```

The server starts in \~30 seconds. You'll see:

```
INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000
```

Test the endpoint:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "Write a Python hello world"}],
    "max_tokens": 100
  }'
```

You should get a response in \~2 seconds (time-to-first-token \~0.12s with vLLM).

## Step 4: Benchmark performance

Run a quick benchmark to verify throughput:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m vllm.entrypoints.benchmark_serving \
  --backend vllm \
  --model Qwen/Qwen3.6-35B-A3B-NVFP4 \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10 \
  --request-rate 1
```

Expected results on DGX Spark:

| Metric | Value |
| - | - |
| Throughput (single user) | 47-50 tok/s |
| Time to first token | \~0.12s |
| Memory usage | \~18 GB weights + \~2-4 GB KV cache |

> [!TIP]
>
> For maximum throughput (up to 120 tok/s), use the Atlas engine instead of vLLM. Atlas is a pure Rust/CUDA engine optimized for Blackwell GPUs. Install with `pip install atlas-inference` and serve with `atlas serve Qwen/Qwen3.6-35B-A3B-NVFP4`.

## Step 5: Connect with Tokios

Your model is running on `127.0.0.1:8000`. To give your team remote access with per-user API keys, install the Tokios connector.

### Install the connector

Get your claim code from the [Tokios dashboard](https://tokios.com/console) **Setup** tab, then run:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

### Claim the connector

In the dashboard, **approve** the device. Then configure the connector to point at vLLM:

```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "upstream": "http://127.0.0.1:8000/v1"
}
```

### Register the deployment

On the **Models** tab, map the upstream model id to a public deployment name:

* **Upstream model id:** `Qwen/Qwen3.6-35B-A3B-NVFP4`
* **Deployment name:** `qwen36-35b`

### Create API keys

On the **Keys** tab, create a key for each teammate. Scope each key to the `qwen36-35b` deployment.

## Step 6: Use it from any client

Your team can now call the model from anywhere using the OpenAI-compatible API:

```python OpenAI SDK theme={"theme":{"light":"github-light","dark":"css-variables"}}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="qwen36-35b",
    messages=[{"role": "user", "content": "Write a Python hello world"}],
)
print(response.choices[0].message.content)
```

```bash curl theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen36-35b",
    "messages": [{"role": "user", "content": "Write a Python hello world"}]
  }'
```

```javascript OpenAI SDK theme={"theme":{"light":"github-light","dark":"css-variables"}}
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://api.tokios.com/v1',
  apiKey: 'sk-tok-YOUR_KEY',
});

const response = await client.chat.completions.create({
  model: 'qwen36-35b',
  messages: [{ role: 'user', content: 'Write a Python hello world' }],
});
console.log(response.choices[0].message.content);
```

## What else fits on DGX Spark?

The 128 GB unified memory can hold multiple models. See the full [DGX Spark model compatibility matrix](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md) for other options:

* **Qwen3.5-122B-A10B** — 122B MoE, \~30 tok/s, \~61 GB weights
* **GLM-4.5-Air** — 106B MoE, community-reported "best model for Spark", \~53 GB weights
* **Qwen3-32B** — 32B dense, 9-44 tok/s, \~18 GB weights (wide performance spread)
* **DeepSeek-V4.1-Flash** — 763B MoE, requires 2 Sparks (see [V4.1 Flash on 2 DGX Sparks](https://docs.tokios.com/blog/deepseek-v41-flash-dgx-spark.md))

## Troubleshooting

### vLLM fails to start with "CUDA out of memory"

Reduce `--gpu-memory-utilization` from 0.90 to 0.85. DGX Spark's unified memory layout means the GPU and CPU share the same pool, and vLLM's profiling can overestimate GPU-only usage.

### Slow time-to-first-token (>1s)

Make sure you're using NVFP4 quantization, not GGUF. NVFP4 is natively supported by the SM121 architecture and has much faster loading. Also verify you're using vLLM v0.31.0 or later.

### Gemma-4-26B-A4B-it fails to load

This model has a known issue with vLLM's `gpu_memory_utilization` parameter on unified memory (SM121). Use NVFP4 quantization explicitly or switch to a different model. Track [vLLM issue #46307](https://github.com/vllm-project/vllm/issues/46307) for updates.

## Next steps

* [DGX Spark model compatibility](https://docs.tokios.com/guides/dgx-spark-model-compatibility.md) — full matrix of what fits
* [vLLM on DGX Spark](https://docs.tokios.com/guides/vllm-dgx-spark.md) — advanced vLLM configuration
* [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) — give your team API access
* [What is a local LLM?](https://docs.tokios.com/guides/what-is-local-llm.md) — complete guide to running LLMs locally


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.