Run Qwen3.6 locally and use it from anywhere

Last verified: 2026-07-21

Qwen3.6 is Qwen’s most widely supported open-weight family: a multimodal mixture-of-experts model that reasons by default, handles up to a 1M-token context, and ships with native tool-calling for agentic coding. Its successor, Qwen3.8-27B, shipped in August 2026 under the same Apache-2.0 license and already has community GGUF builds, but no Ollama tag — so Qwen3.6 remains the easiest verified local setup. It comes in two sizes — a 35B-A3B MoE (35B total, 3B active) and a 27B dense model. Run whichever fits your hardware locally, then pair the Tokios connector to call it from any machine, teammate, or hosted agent through https://api.tokios.com — no inbound ports, no VPN.

Already running Qwen3.6 locally? Register it with Tokios to reach it as https://api.tokios.com from any client.

Looking for Qwen 3.5? Qwen3.6 is the generation that replaced it. Alibaba’s dense Qwen3.6-27B outperforms the previous-generation Qwen3.5-397B-A17B MoE on several agentic coding tasks despite being far smaller — so if you were about to pull Qwen3.5, this is the model to run instead.

Run it locally with Ollama

Qwen3.6 is in the Ollama library. Pull and run the default tag (the 35B-A3B MoE):

ollama run qwen3.6

For the smaller dense model, run ollama run qwen3.6:27b. See the Ollama library page for the full tag list — this page covers connecting it, not every runtime detail.

Ollama’s default context window is small (2048 tokens) — far below Qwen3.6’s native 256K. Raise num_ctx in your Modelfile or client request to use more of its context. Qwen recommends keeping at least 128K to preserve reasoning quality on complex tasks.

Variants & tags

Every value below is from the model card and the Ollama library. Tags and download sizes change as new quantizations ship — confirm against ollama.com/library/qwen3.6 before you standardize on one.

Property	`35B-A3B` (MoE)	`27B` (dense)
Ollama tag	`qwen3.6:35b` (alias `:latest`)	`qwen3.6:27b`
Download size (`Q4_K_M`)	~24 GB	~17 GB
Parameters	35B total / 3B active (256 experts, 8 routed + 1 shared)	27B dense
Context window	262,144 native, extensible to ~1M	262,144 native
Modality	Text + image + video	Text + image + video
Reasoning	Thinking mode on by default	Thinking mode on by default
License	Apache-2.0	Apache-2.0
Hugging Face	`Qwen/Qwen3.6-35B-A3B`	`Qwen/Qwen3.6-27B`

Sources: ollama.com/library/qwen3.6, Qwen/Qwen3.6-35B-A3B model card, Qwen/Qwen3.6-27B.

The 35B-A3B is a mixture-of-experts model with only 3B active parameters per token, so it decodes about as fast as a 3B model while reasoning like a much larger one — but all 35B parameters still have to fit in memory. Don’t size your GPU off the active-parameter count alone. See how inference works for why.

Quantization & formats

Ollama’s default tags are GGUF Q4_K_M. The library also publishes -q8_0, -bf16, -nvfp4 (NVIDIA Blackwell), -mxfp8, -mlx (Apple Silicon), and -mtp-* (multi-token-prediction speculative-decoding) variants for both sizes. Which format you can run depends on your hardware — see Quantization formats and the hardware they need for the full matrix, and what quantization is for how to read the Q4_K_M-style names. Unlike Gemma, Qwen doesn’t publish official quantization-aware-trained (QAT) checkpoints — Qwen’s tags above are post-training quantizations. If you want QAT-grade quality at 4-bit, fine-tune your own with a tool like Unsloth; see quantization-aware training for when that’s worth it.

KV-cache quantization

The KV cache is the per-token memory a model builds as it reads your prompt and generates a reply — and it grows with every token of context. For a long-context model like Qwen3.6 (262K native, extensible to ~1M), the KV cache can rival or exceed the weight footprint at full context. Quantizing the KV cache is a separate lever from weight quantization that cuts that memory cost roughly in half.

Enable FP8 KV cache

On vLLM, add --kv-cache-dtype fp8 to your serve command:

vllm serve Qwen/Qwen3.6-35B-A3B \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

On SGLang, the equivalent flag is --kv-cache-dtype fp8:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.6-35B-A3B \
  --context-length 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Ollama and LM Studio manage the KV cache internally and do not expose a --kv-cache-dtype flag. If you need FP8 KV cache, use vLLM or SGLang as the serving backend.

What you get

Metric	FP16 KV cache	FP8 KV cache
Bytes per cache entry	2	1
Memory reduction	—	~50%
Quality impact	Baseline	2% degradation on standard benchmarks

The practical payoff is longer context on the same hardware. A Qwen3.6 deployment that fits 32K tokens with FP16 KV cache can typically serve 64K or more with FP8 KV cache in the same memory budget — because the cache takes half the space per token. On memory-constrained hardware like a single DGX Spark, that difference can be the gap between a usable long-context deployment and an OOM. For hardware-specific tuning, see DGX Spark model compatibility.

FP8 KV cache is a quality-for-memory trade-off, not a free win. On most tasks — chat, summarization, code generation — the 2% quality loss is negligible. If your workload is sensitive to precision (long-form reasoning, mathematical proofs), benchmark with and without FP8 KV cache on your actual prompts before committing.

Hardware fit

Comfort at roughly Q4 for a modest context length. Treat every entry as an estimate — the ground truth is the actual GGUF size plus the KV cache for your context. See what size model fits your GPU? for the sizing math.

Target	`27B` dense (~17 GB)	`35B-A3B` MoE (~24 GB)
RTX 3090 / 4090 (24 GB)	Comfortable	Tight — reduce context
RTX 5090 (32 GB, Blackwell)	Comfortable; `nvfp4` tag available	Comfortable; `nvfp4` tag available
RTX 6000 Ada / A6000 (48 GB)	Comfortable, long context	Comfortable, long context
RTX 5080 / 4080 (16 GB)	Tight — short context	Won’t fit at `Q4`
Apple Silicon (36 GB+ unified)	Comfortable; `mlx` tag available	Comfortable; `mlx` tag available
AMD Strix Halo (ROCm/Vulkan, ≤96 GB)	Comfortable	Comfortable
DGX Spark (128 GB unified, Blackwell)	Comfortable (~14–15 GB at NVFP4, estimated 40–55 tok/s)	Excellent fit (~18 GB at NVFP4, estimated 47–50 tok/s; up to 120 tok/s with Atlas)

Can a DGX Spark run Qwen3.6?

Yes, and it is one of the best matches for this family. The dense 27B lands near 14 GB at NVFP4 and the 35B-A3B MoE near 18 GB, against 128 GB of unified memory. Because the weights claim so little of the pool, the rest is available for KV cache — which makes the 262K native context practical here rather than theoretical. Prefer the MoE for throughput: only 3B parameters are active per token, so it decodes far faster than its total size suggests. See DGX Spark model compatibility for tuning and measured behaviour.

Can Strix Halo run Qwen3.6?

Yes. AMD Strix Halo can allocate up to 96 GB to the GPU, so both variants load with room left for long context. Capacity is not the limit on this machine — memory bandwidth is. Decode trails a discrete GPU holding the same weights, and prompt processing trails it further, so favour the 35B-A3B MoE where its small active-parameter count does the most good. Running vLLM on Strix Halo and Strix Halo model compatibility cover the ROCm and Vulkan paths.

Can a 24 GB GPU run Qwen3.6?

Yes for the dense 27B at Q4_K_M, which needs about 17 GB and leaves roughly 7 GB for KV cache — comfortable at everyday context lengths. The 35B-A3B MoE needs about 24 GB, so on a 24 GB card it is tight: shorten the context or move to a 32 GB card. On 16 GB, the dense 27B runs only at short context, and the MoE does not fit at Q4. Offloading to system RAM works but costs throughput — see memory and offloading.

Fastest way to run Qwen3.6 locally

For the most tokens per second on modest hardware, reach for the 35B-A3B MoE: only 3B of its 35B parameters are active per token, so it decodes about as fast as a 3B model while reasoning like a much larger one — provided all 35B fit in memory (~24 GB at Q4_K_M). If you specifically want the dense 27B, pull Q4_K_M on Ollama or LM Studio (~17 GB); on Apple Silicon the -mlx tag runs fastest. Raise num_ctx only as far as you actually need — a shorter context leaves more memory for weights and speeds up decode.

# Fastest general pick: the MoE (only ~3B active per token)
ollama run qwen3.6

# Dense 27B at Q4_K_M
ollama run qwen3.6:27b

Run it — beginner to advanced

Ollama (easiest)
LM Studio (GUI)
vLLM (advanced)
SGLang (advanced)

ollama run qwen3.6

Leave Ollama listening on the default http://127.0.0.1:11434. Raise num_ctx to use more context.

Search for Qwen3.6 in LM Studio’s model catalog, download a GGUF quant that fits your VRAM (start with Q4_K_M), then start LM Studio’s local server. It exposes an OpenAI-compatible API on http://127.0.0.1:1234/v1.

Qwen3.6 needs its reasoning and tool-call parsers set explicitly. For tool use:

vllm serve Qwen/Qwen3.6-35B-A3B \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Add --tensor-parallel-size <N> to shard across GPUs. Flags are from the model card’s vLLM section.

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.6-35B-A3B \
  --context-length 262144 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Add --tp-size <N> for tensor parallelism. Flags are from the model card’s SGLang section.

Capabilities

Reasoning: Qwen3.6 runs in thinking mode by default, emitting a <think>…</think> block before its answer. To get a direct response, disable it with "chat_template_kwargs": {"enable_thinking": false} in your request (or "enable_thinking": false on Alibaba Cloud Model Studio). It also adds a “thinking preservation” option that retains reasoning context across turns.
Native tool-calling: Yes. On vLLM/SGLang, set --tool-call-parser qwen3_coder (with --reasoning-parser qwen3); Ollama applies the template for you. Built for agentic coding — repository-level reasoning and frontend workflows.
Vision: Multimodal — accepts text, images, and video.
Long context: 262,144 tokens natively, extensible to ~1M. Keep at least 128K for complex reasoning tasks.

Connect it with Tokios

Once Qwen3.6 is running locally, pair the Tokios connector so you can reach it as https://api.tokios.com from Claude Code, Cline, or any other compatible client — instead of only from localhost:11434.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing to get a one-time claim code (TKS-XXXX-XXXX). Install and run the connector on the machine running Ollama:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Then approve the device on the Setup or Connectors tab.

Point the connector at Ollama's /v1 API

Ollama exposes an OpenAI-compatible surface at /v1. For a single-model setup, the console’s Setup wizard fills this in automatically as http://127.0.0.1:11434/v1. For multiple models, set it explicitly in tokios-connector.json — see Connector config.

In the Models tab, register the model and choose a public deployment name, for example qwen3.6-tunnel. This name — not Ollama’s local model id — is what clients send in the model field.

Create an API key

In the Keys tab, click Create key. Copy the sk-tok-… key now — it’s shown only once.

Call it from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.6-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Refactor this function and explain your reasoning."}]
  }'

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model qwen3.6-tunnel

The model field is always your registered deployment name (qwen3.6-tunnel above), never Ollama’s local model id. See Use Claude Code and Use Cline for full client setup.

Caveats

Qwen3.6 emits <think>…</think> reasoning by default. If a downstream client shows the raw thinking or you want lower latency, disable thinking as shown above.
Tool-call behavior depends on the runtime applying the qwen3_coder parser. Ollama handles this for you; if you self-host with vLLM/SGLang, set the flags above or tool calls won’t parse.
The 35B-A3B MoE needs all 35B parameters in memory even though only 3B are active per token — size your hardware off the full download, not the active count.

New same-class alternative. If you are weighing the 27B dense build for agent work, Meta’s Muse Glimmer 30B is now a third option at that tier — also dense, also Apache 2.0, and trained specifically for tool calling and multi-step task completion. Meta benchmarks it against Qwen3.6 27B directly and its numbers favour Muse Glimmer, but they are vendor-run with no independent reproduction yet. Qwen3.6 has the longer track record and the wider variant range; test both before you commit an agent workflow to either.

Quickstart

Pair a connector and register your first model end to end.

What size model fits your GPU?

Match Qwen3.6 to your available VRAM before you pull it.

Quantization & hardware

Which of Qwen3.6’s formats — GGUF, NVFP4, MLX — runs on your hardware.

Which local model for which task?

See how Qwen3.6 compares to other local models by task.

Get Started

Models

Concepts

Reference

Run Qwen3.6 locally and use it from anywhere

Run it locally with Ollama

Variants & tags

Quantization & formats

KV-cache quantization

Enable FP8 KV cache

What you get

Hardware fit

Can a DGX Spark run Qwen3.6?

Can Strix Halo run Qwen3.6?

Can a 24 GB GPU run Qwen3.6?

Fastest way to run Qwen3.6 locally

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

​Run it locally with Ollama

​Variants & tags

​Quantization & formats

​KV-cache quantization

​Enable FP8 KV cache

​What you get

​Hardware fit

​Can a DGX Spark run Qwen3.6?

​Can Strix Halo run Qwen3.6?

​Can a 24 GB GPU run Qwen3.6?

​Fastest way to run Qwen3.6 locally

​Run it — beginner to advanced

​Capabilities

​Connect it with Tokios

​Call it from anywhere

​Caveats

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

Run it locally with Ollama

Variants & tags

Quantization & formats

KV-cache quantization

Enable FP8 KV cache

What you get

Hardware fit

Can a DGX Spark run Qwen3.6?

Can Strix Halo run Qwen3.6?

Can a 24 GB GPU run Qwen3.6?

Fastest way to run Qwen3.6 locally

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats