Run Qwen3.6-35B-A3B on RTX 5090

Qwen3.6-35B-A3B is the ideal model for RTX 5090. It’s a 35B-parameter mixture-of-experts model with only 3B active parameters per token. At Q6_K quantization, the model uses ~28 GB — fitting comfortably in the RTX 5090’s 32 GB GDDR7 with room for KV cache. With native FP4 tensor cores, you get ~90 tok/s throughput. This guide walks you through downloading the model, serving it with vLLM, and exposing it through Tokios for team access. Last verified: 2026-10-24

What you’ll get

Model: Qwen3.6-35B-A3B (35B total, 3B active per token)
Hardware: Single NVIDIA RTX 5090 (32 GB GDDR7)
Throughput: ~90 tok/s with vLLM
Context: 32K+ tokens
Memory usage: ~28 GB for weights, leaving ~4 GB for KV cache
Use case: High-speed coding assistant, chat, JSON extraction, agentic workflows

Prerequisites

NVIDIA RTX 5090 (32 GB) with CUDA 12.x installed
850W+ power supply (RTX 5090 draws 575W)
Internet connection for downloading model weights (~28 GB)
Tokios account (free tier works)
20 minutes

RTX 5090 availability and pricing. NVIDIA has confirmed RTX 5090 is not discontinued — it remains in production. However, GDDR7 shortages have driven street prices above $4,000 (vs $1,999 MSRP). If you can buy at or near MSRP, do not wait. Consider used RTX 4090 (~$1,600-2,000, same 24 GB VRAM) as an alternative — it runs the same model at ~65 tok/s instead of ~90 tok/s.

Step 1: Install vLLM

Install vLLM v0.31.0 or later with CUDA support:

pip install --upgrade vllm

Verify the installation and CUDA detection:

vllm --version
nvidia-smi

You should see the RTX 5090 listed with 32 GB VRAM and CUDA 12.x drivers.

Step 2: Download the model

The RTX 5090 has native FP4 tensor cores (Blackwell architecture). Use the NVFP4 quantization for best performance:

huggingface-cli download Qwen/Qwen3.6-35B-A3B-NVFP4

The download is ~18 GB. On a typical consumer internet connection, this takes 10-20 minutes.

If you prefer GGUF format for llama.cpp or Ollama, use Qwen/Qwen3.6-35B-A3B-GGUF with Q6_K quantization (~28 GB). However, NVFP4 is recommended for RTX 5090 because it’s natively supported by the Blackwell FP4 tensor cores and delivers 20-40% better performance than GGUF.

Step 3: Serve the model with vLLM

Start the vLLM server with the NVFP4 model:

vllm serve Qwen/Qwen3.6-35B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85

The server starts in ~20 seconds. You’ll see:

INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000

Test the endpoint:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "Write a Python hello world"}],
    "max_tokens": 100
  }'

You should get a response in ~1 second.

Step 4: Benchmark performance

Run a quick benchmark to verify throughput:

python -m vllm.entrypoints.benchmark_serving \
  --backend vllm \
  --model Qwen/Qwen3.6-35B-A3B-NVFP4 \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10 \
  --request-rate 1

Expected results on RTX 5090:

Metric	Value
Throughput (single user)	~90 tok/s
Time to first token	~0.1s
Memory usage	~28 GB weights + ~3-4 GB KV cache
VRAM headroom	~0-1 GB (tight fit)

The RTX 5090 is ~39% faster than RTX 4090 on this model (90 vs 65 tok/s) thanks to native FP4 tensor cores and higher memory bandwidth (1,792 vs 1,008 GB/s). If you’re upgrading from RTX 4090, expect a significant speedup.

Step 5: Connect with Tokios

Your model is running on 127.0.0.1:8000. To give your team remote access with per-user API keys, install the Tokios connector.

Install the connector

Get your claim code from the Tokios dashboard Setup tab, then run:

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Windows (PowerShell)

irm https://tokios.com/downloads/install/windows.ps1 | iex

Claim the connector

In the dashboard, approve the device. Then configure the connector to point at vLLM:

{
  "upstream": "http://127.0.0.1:8000/v1"
}

Register the deployment

On the Models tab, map the upstream model id to a public deployment name:

Upstream model id: Qwen/Qwen3.6-35B-A3B-NVFP4
Deployment name: qwen36-35b-rtx

Create API keys

On the Keys tab, create a key for each teammate. Scope each key to the qwen36-rtx deployment.

Step 6: Use it from any client

Your team can now call the model from anywhere using the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="qwen36-35b-rtx",
    messages=[{"role": "user", "content": "Write a Python hello world"}],
)
print(response.choices[0].message.content)

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen36-35b-rtx",
    "messages": [{"role": "user", "content": "Write a Python hello world"}]
  }'

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://api.tokios.com/v1',
  apiKey: 'sk-tok-YOUR_KEY',
});

const response = await client.chat.completions.create({
  model: 'qwen36-35b-rtx',
  messages: [{ role: 'user', content: 'Write a Python hello world' }],
});
console.log(response.choices[0].message.content);

What else fits on RTX 5090?

The 32 GB VRAM can hold multiple model sizes. See the full RTX 50 series guide for other options:

Qwen3-70B at Q4 — ~40 GB, tight fit with KV cache, ~23 tok/s
Qwen3.5-122B-A10B at Q4 — ~70 GB, does NOT fit (need 2x RTX 5090)
Llama 3.1 32B at Q4 — ~18 GB, comfortable fit, ~45 tok/s
Multiple 14B models simultaneously — each ~8 GB, run 2-3 in parallel

Troubleshooting

vLLM fails to start with “CUDA out of memory”

Reduce --gpu-memory-utilization from 0.85 to 0.80. The RTX 5090’s 32 GB is a tight fit for the 28 GB model, and vLLM’s profiling can overestimate available memory.

Slow throughput (<70 tok/s)

Make sure you’re using NVFP4 quantization, not GGUF. NVFP4 is natively supported by the Blackwell FP4 tensor cores and delivers 20-40% better performance. Also verify you’re using vLLM v0.31.0 or later.

System instability or crashes

The RTX 5090 draws 575W. Ensure your PSU can handle it (850W+ recommended) and your case has adequate cooling. Monitor temperatures with nvidia-smi dmon during inference.

KV cache runs out at long context

Reduce --max-model-len from 32768 to 16384 or 8192. The 32 GB VRAM is a tight fit, and long contexts consume significant KV cache memory.

Next steps

RTX 50 series consumer guide — full RTX 5090/5080/5070 comparison
Team access to local models — give your team API access
What is a local LLM? — complete guide to running LLMs locally
Model fit calculator — check what fits on your hardware

Getting started with local LLMs

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Run Qwen3.6-35B-A3B on RTX 5090

What you’ll get

Prerequisites

Step 1: Install vLLM

Step 2: Download the model

Step 3: Serve the model with vLLM

Step 4: Benchmark performance

Step 5: Connect with Tokios

Install the connector

Claim the connector

Register the deployment

Create API keys

Step 6: Use it from any client

What else fits on RTX 5090?

Troubleshooting

vLLM fails to start with “CUDA out of memory”

Slow throughput (<70 tok/s)

System instability or crashes

KV cache runs out at long context

Next steps

​What you’ll get

​Prerequisites

​Step 1: Install vLLM

​Step 2: Download the model

​Step 3: Serve the model with vLLM

​Step 4: Benchmark performance

​Step 5: Connect with Tokios

​Install the connector

​Claim the connector

​Register the deployment

​Create API keys

​Step 6: Use it from any client

​What else fits on RTX 5090?

​Troubleshooting

​vLLM fails to start with “CUDA out of memory”

​Slow throughput (<70 tok/s)

​System instability or crashes

​KV cache runs out at long context

​Next steps

What you’ll get

Prerequisites

Step 1: Install vLLM

Step 2: Download the model

Step 3: Serve the model with vLLM

Step 4: Benchmark performance

Step 5: Connect with Tokios

Install the connector

Claim the connector

Register the deployment

Create API keys

Step 6: Use it from any client

What else fits on RTX 5090?

Troubleshooting

vLLM fails to start with “CUDA out of memory”

Slow throughput (<70 tok/s)

System instability or crashes

KV cache runs out at long context

Next steps