> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Run Qwen3.6-35B-A3B on RTX 5090

> Step-by-step guide to running Qwen3.6-35B-A3B (35B MoE, 3B active) on NVIDIA RTX 5090 (32 GB) with vLLM and Tokios. Achieve ~90 tok/s with NVFP4 quantization.

Qwen3.6-35B-A3B is the ideal model for RTX 5090. It's a 35B-parameter mixture-of-experts model with only 3B active parameters per token. At Q6\_K quantization, the model uses \~28 GB — fitting comfortably in the RTX 5090's 32 GB GDDR7 with room for KV cache. With native FP4 tensor cores, you get \~90 tok/s throughput.

This guide walks you through downloading the model, serving it with vLLM, and exposing it through Tokios for team access.

Last verified: 2026-10-24

## What you'll get

* **Model:** Qwen3.6-35B-A3B (35B total, 3B active per token)
* **Hardware:** Single NVIDIA RTX 5090 (32 GB GDDR7)
* **Throughput:** \~90 tok/s with vLLM
* **Context:** 32K+ tokens
* **Memory usage:** \~28 GB for weights, leaving \~4 GB for KV cache
* **Use case:** High-speed coding assistant, chat, JSON extraction, agentic workflows

## Prerequisites

* NVIDIA RTX 5090 (32 GB) with CUDA 12.x installed
* 850W+ power supply (RTX 5090 draws 575W)
* Internet connection for downloading model weights (\~28 GB)
* Tokios account (free tier works)
* 20 minutes

> [!WARNING]
>
> **RTX 5090 availability and pricing.** NVIDIA has confirmed RTX 5090 is **not discontinued** — it remains in production. However, GDDR7 shortages have driven street prices above $4,000 (vs $1,999 MSRP). If you can buy at or near MSRP, do not wait. Consider **used RTX 4090** (\~\$1,600-2,000, same 24 GB VRAM) as an alternative — it runs the same model at \~65 tok/s instead of \~90 tok/s.

## Step 1: Install vLLM

Install vLLM v0.31.0 or later with CUDA support:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
pip install --upgrade vllm
```

Verify the installation and CUDA detection:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm --version
nvidia-smi
```

You should see the RTX 5090 listed with 32 GB VRAM and CUDA 12.x drivers.

## Step 2: Download the model

The RTX 5090 has native FP4 tensor cores (Blackwell architecture). Use the NVFP4 quantization for best performance:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
huggingface-cli download Qwen/Qwen3.6-35B-A3B-NVFP4
```

The download is \~18 GB. On a typical consumer internet connection, this takes 10-20 minutes.

> [!NOTE]
>
> If you prefer GGUF format for llama.cpp or Ollama, use `Qwen/Qwen3.6-35B-A3B-GGUF` with Q6\_K quantization (\~28 GB). However, NVFP4 is recommended for RTX 5090 because it's natively supported by the Blackwell FP4 tensor cores and delivers 20-40% better performance than GGUF.

## Step 3: Serve the model with vLLM

Start the vLLM server with the NVFP4 model:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve Qwen/Qwen3.6-35B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85
```

The server starts in \~20 seconds. You'll see:

```
INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000
```

Test the endpoint:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "Write a Python hello world"}],
    "max_tokens": 100
  }'
```

You should get a response in \~1 second.

## Step 4: Benchmark performance

Run a quick benchmark to verify throughput:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
python -m vllm.entrypoints.benchmark_serving \
  --backend vllm \
  --model Qwen/Qwen3.6-35B-A3B-NVFP4 \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10 \
  --request-rate 1
```

Expected results on RTX 5090:

| Metric | Value |
| - | - |
| Throughput (single user) | \~90 tok/s |
| Time to first token | \~0.1s |
| Memory usage | \~28 GB weights + \~3-4 GB KV cache |
| VRAM headroom | \~0-1 GB (tight fit) |

> [!TIP]
>
> The RTX 5090 is \~39% faster than RTX 4090 on this model (90 vs 65 tok/s) thanks to native FP4 tensor cores and higher memory bandwidth (1,792 vs 1,008 GB/s). If you're upgrading from RTX 4090, expect a significant speedup.

## Step 5: Connect with Tokios

Your model is running on `127.0.0.1:8000`. To give your team remote access with per-user API keys, install the Tokios connector.

### Install the connector

Get your claim code from the [Tokios dashboard](https://tokios.com/console) **Setup** tab, then run:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX
```

```bash Windows (PowerShell) theme={"theme":{"light":"github-light","dark":"css-variables"}}
irm https://tokios.com/downloads/install/windows.ps1 | iex
```

### Claim the connector

In the dashboard, **approve** the device. Then configure the connector to point at vLLM:

```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "upstream": "http://127.0.0.1:8000/v1"
}
```

### Register the deployment

On the **Models** tab, map the upstream model id to a public deployment name:

* **Upstream model id:** `Qwen/Qwen3.6-35B-A3B-NVFP4`
* **Deployment name:** `qwen36-35b-rtx`

### Create API keys

On the **Keys** tab, create a key for each teammate. Scope each key to the `qwen36-rtx` deployment.

## Step 6: Use it from any client

Your team can now call the model from anywhere using the OpenAI-compatible API:

```python OpenAI SDK theme={"theme":{"light":"github-light","dark":"css-variables"}}
from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="qwen36-35b-rtx",
    messages=[{"role": "user", "content": "Write a Python hello world"}],
)
print(response.choices[0].message.content)
```

```bash curl theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen36-35b-rtx",
    "messages": [{"role": "user", "content": "Write a Python hello world"}]
  }'
```

```javascript OpenAI SDK theme={"theme":{"light":"github-light","dark":"css-variables"}}
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://api.tokios.com/v1',
  apiKey: 'sk-tok-YOUR_KEY',
});

const response = await client.chat.completions.create({
  model: 'qwen36-35b-rtx',
  messages: [{ role: 'user', content: 'Write a Python hello world' }],
});
console.log(response.choices[0].message.content);
```

## What else fits on RTX 5090?

The 32 GB VRAM can hold multiple model sizes. See the full [RTX 50 series guide](https://docs.tokios.com/guides/rtx-50-series-consumer.md) for other options:

* **Qwen3-70B at Q4** — \~40 GB, tight fit with KV cache, \~23 tok/s
* **Qwen3.5-122B-A10B at Q4** — \~70 GB, does NOT fit (need 2x RTX 5090)
* **Llama 3.1 32B at Q4** — \~18 GB, comfortable fit, \~45 tok/s
* **Multiple 14B models simultaneously** — each \~8 GB, run 2-3 in parallel

## Troubleshooting

### vLLM fails to start with "CUDA out of memory"

Reduce `--gpu-memory-utilization` from 0.85 to 0.80. The RTX 5090's 32 GB is a tight fit for the 28 GB model, and vLLM's profiling can overestimate available memory.

### Slow throughput (\<70 tok/s)

Make sure you're using NVFP4 quantization, not GGUF. NVFP4 is natively supported by the Blackwell FP4 tensor cores and delivers 20-40% better performance. Also verify you're using vLLM v0.31.0 or later.

### System instability or crashes

The RTX 5090 draws 575W. Ensure your PSU can handle it (850W+ recommended) and your case has adequate cooling. Monitor temperatures with `nvidia-smi dmon` during inference.

### KV cache runs out at long context

Reduce `--max-model-len` from 32768 to 16384 or 8192. The 32 GB VRAM is a tight fit, and long contexts consume significant KV cache memory.

## Next steps

* [RTX 50 series consumer guide](https://docs.tokios.com/guides/rtx-50-series-consumer.md) — full RTX 5090/5080/5070 comparison
* [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) — give your team API access
* [What is a local LLM?](https://docs.tokios.com/guides/what-is-local-llm.md) — complete guide to running LLMs locally
* [Model fit calculator](https://docs.tokios.com/guides/model-fit.md) — check what fits on your hardware


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.