FreeToken: MoE inference on consumer NVIDIA GPUs

FreeToken is an open-source inference engine from UC Berkeley that makes it practical to run frontier-scale Mixture-of-Experts (MoE) models on consumer NVIDIA GPUs. Where traditional serving stacks require you to fit the entire model in VRAM, FreeToken uses dynamic CPU-GPU co-scheduling to stream expert weights from system memory as needed — letting you run a 35B-parameter model on an 8GB GPU at ~39 tokens/s.

Last verified: 2026-10-20

What FreeToken solves

MoE models like DeepSeek-V4-Flash (284B total, 8B active) or Qwen3.6-35B (30.5B total, 3.3B active) generate tokens at the speed of their active parameters — but they need memory for all parameters. A 35B MoE at 4-bit quantization needs ~18 GB of memory, which doesn’t fit on an 8GB GPU. Traditional offloading (llama.cpp, Ollama) moves weights to system RAM statically, which works but is slow — PCIe bandwidth becomes the bottleneck. FreeToken takes a different approach:

Dynamic co-scheduling — the engine predicts which experts you’ll need next and streams them from CPU to GPU ahead of time, overlapping transfer with compute
Double buffering — while one set of experts runs on GPU, the next set transfers from CPU, hiding latency
Bandwidth-adaptive execution — FreeToken measures your PCIe bandwidth and adjusts the scheduling policy to match

The result: 3-4x faster decode than static offloading on the same hardware.

Supported models

FreeToken currently supports three MoE model families:

Model	Total params	Active params	Memory at 4-bit	Status
Qwen3.6-35B-A3B	30.5B	3.3B	~18 GB	Stable
DeepSeek-V4-Flash	284B	8B (in) / 16B (out)	~170 GB	Stable
GLM-5.2	753B	~40B	~452 GB	Experimental

The 35B model fits on a single 24GB GPU at 4-bit. The 284B and 753B models require system RAM offloading — FreeToken’s dynamic scheduling makes this practical on a gaming desktop with 64-128 GB of system RAM.

Hardware requirements

Minimum:

NVIDIA RTX 30/40/50 series GPU (8GB+ VRAM)
32 GB system RAM (64 GB+ recommended for 284B+ models)
PCIe 3.0 x16 or better (PCIe 4.0/5.0 preferred)
CUDA 12.x

Not supported:

Apple Silicon (use MLX instead — see MLX distributed inference)
AMD GPUs
Pre-Pascal NVIDIA GPUs (no CUDA compute capability 6.0+)

FreeToken is optimized for NVIDIA’s consumer GPU lineup. If you’re on a Mac with Apple Silicon, the MLX framework with distributed inference across multiple Macs is the equivalent path for running 800B+ models locally.

Installation

Install FreeToken with uv (recommended) or pip:

# Using uv (faster, isolated environment)
uv pip install "freetoken[accel]"

# Using pip
pip install "freetoken[accel]"

The [accel] extra installs CUDA kernels and optimization dependencies. Without it, FreeToken falls back to a slower reference implementation.

Quick start

Serve Qwen3.6-35B on a 24GB GPU

# Download the model (4-bit quantization)
freetoken download qwen3.6-35b-a3b:q4

# Start the server
freetoken serve qwen3.6-35b-a3b:q4 --port 8000

FreeToken exposes an OpenAI-compatible API at http://127.0.0.1:8000/v1. Test it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.6-35b-a3b:q4",
    "messages": [{"role": "user", "content": "Explain MoE routing in one paragraph."}],
    "max_tokens": 256
  }'

Expected performance on an RTX 4090 (24GB):

Decode speed: ~39 tokens/s
Time to first token: ~1.2s (4K prompt)
Memory usage: ~18 GB VRAM + ~2 GB system RAM

Serve DeepSeek-V4-Flash with CPU offloading

For the 284B model, you’ll need 64+ GB of system RAM and a GPU with at least 8GB VRAM:

# Download (this is ~170 GB at 4-bit)
freetoken download deepseek-v4-flash:q4

# Serve with CPU offloading enabled
freetoken serve deepseek-v4-flash:q4 \
  --port 8000 \
  --cpu-offload \
  --offload-ratio 0.8

The --offload-ratio 0.8 tells FreeToken to keep 20% of experts in GPU memory and stream the remaining 80% from system RAM. Adjust based on your VRAM:

GPU VRAM	Offload ratio	Expected decode speed
8 GB	0.9	~12 tokens/s
12 GB	0.85	~18 tokens/s
16 GB	0.75	~26 tokens/s
24 GB	0.6	~35 tokens/s

Connect to Tokios

Once FreeToken is running on localhost:8000, point the Tokios connector at it:

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8000/v1"
  }
}

Register the model in the Tokios dashboard and you get an authenticated API endpoint you can call from anywhere. See Supported backends for the full connector setup flow.

Performance tuning

Optimize for throughput

If you’re serving multiple concurrent requests, enable continuous batching:

freetoken serve qwen3.6-35b-a3b:q4 \
  --port 8000 \
  --batching \
  --max-batch-size 8 \
  --max-wait-ms 10

This batches up to 8 requests together, amortizing the expert-loading cost across multiple users. Latency per request increases slightly, but total throughput improves.

Optimize for latency

For interactive use where you want the fastest possible first token:

freetoken serve qwen3.6-35b-a3b:q4 \
  --port 8000 \
  --prefill-priority high \
  --disable-batching

This prioritizes prefill (prompt processing) over decode, reducing time-to-first-token at the cost of lower overall throughput.

Monitor resource usage

FreeToken includes a built-in metrics endpoint:

curl http://127.0.0.1:8000/metrics

Returns:

GPU memory usage
System RAM usage
PCIe transfer rate (GB/s)
Decode throughput (tokens/s)
Active expert count per token

Use this to tune your offload ratio — if PCIe transfer rate is low and decode speed is suffering, reduce the offload ratio to keep more experts in GPU memory.

FreeToken vs other backends

Backend	MoE support	35B on 8GB GPU	Decode speed	Setup complexity
FreeToken	Native, optimized	Yes	~39 tok/s (RTX 4090)	Medium (Python package)
Ollama	Static offloading	Yes	~12 tok/s (RTX 4090)	Low (one command)
llama.cpp	Static offloading	Yes	~15 tok/s (RTX 4090)	Medium (build from source)
vLLM	GPU-only	No (needs 24GB+)	~45 tok/s (RTX 4090, full fit)	High (GPU server setup)

FreeToken’s advantage is the dynamic scheduling — it’s 2.5-3x faster than static offloading on the same hardware when the model doesn’t fully fit in VRAM. If your model does fit entirely in VRAM, vLLM is slightly faster, but FreeToken remains competitive.

Troubleshooting

”CUDA out of memory” even with CPU offloading

Your --offload-ratio is too low. Increase it to stream more experts from CPU:

freetoken serve deepseek-v4-flash:q4 --cpu-offload --offload-ratio 0.95

Decode speed is slower than expected

Check PCIe bandwidth:

freetoken benchmark --model qwen3.6-35b-a3b:q4 --test pcie

If PCIe transfer rate is below 10 GB/s, you may have a bottleneck (older PCIe generation, or the GPU is in a slower slot). FreeToken’s dynamic scheduling depends on fast CPU-GPU transfers.

Model download is slow

FreeToken downloads from Hugging Face. If you’re in a region with slow HF access, use a mirror:

export HF_ENDPOINT=https://hf-mirror.com
freetoken download qwen3.6-35b-a3b:q4

MLX distributed inference

If you’re on Apple Silicon and want to run 800B+ models locally, MLX’s distributed inference is the equivalent of FreeToken’s CPU-GPU co-scheduling — but across multiple Macs. MLX distributed (announced October 2026) uses RDMA over Thunderbolt to cluster Mac Studios or Mac Pros. A 4× M5 Ultra cluster achieves 36.73 tokens/s on Muse Spark Open (816B parameters). Setup:

# On each Mac
pip install mlx-lm

# Start the cluster
mlx_lm.distributed \
  --model muse-spark-open:q4 \
  --nodes 4 \
  --parallelism tensor

See Apple ML Research for the full technical details.

Next steps

Supported backends

Compare FreeToken to other local inference backends.

MoE concepts

Understand how Mixture-of-Experts routing works and why memory scales with total parameters.

Model fit

Calculate whether a model and quantization fit your hardware.

Connect to Tokios

Turn your local FreeToken server into an authenticated API endpoint.

Sources

FreeToken GitHub repository — installation, supported models, and technical details.
FreeToken paper (arXiv) — bandwidth-adaptive execution and dynamic co-scheduling.
InfoQ coverage — practical benchmarks and hardware requirements.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

FreeToken: MoE inference on consumer NVIDIA GPUs

What FreeToken solves

Supported models

Hardware requirements

Installation

Quick start

Serve Qwen3.6-35B on a 24GB GPU

Serve DeepSeek-V4-Flash with CPU offloading

Connect to Tokios

Performance tuning

Optimize for throughput

Optimize for latency

Monitor resource usage

FreeToken vs other backends

Troubleshooting

”CUDA out of memory” even with CPU offloading

Decode speed is slower than expected

Model download is slow

MLX distributed inference

Next steps

Supported backends

MoE concepts

Model fit

Connect to Tokios

Sources

​What FreeToken solves

​Supported models

​Hardware requirements

​Installation

​Quick start

​Serve Qwen3.6-35B on a 24GB GPU

​Serve DeepSeek-V4-Flash with CPU offloading

​Connect to Tokios

​Performance tuning

​Optimize for throughput

​Optimize for latency

​Monitor resource usage

​FreeToken vs other backends

​Troubleshooting

​”CUDA out of memory” even with CPU offloading

​Decode speed is slower than expected

​Model download is slow

​MLX distributed inference

​Next steps

Supported backends

MoE concepts

Model fit

Connect to Tokios

​Sources

What FreeToken solves

Supported models

Hardware requirements

Installation

Quick start

Serve Qwen3.6-35B on a 24GB GPU

Serve DeepSeek-V4-Flash with CPU offloading

Connect to Tokios

Performance tuning

Optimize for throughput

Optimize for latency

Monitor resource usage

FreeToken vs other backends

Troubleshooting

”CUDA out of memory” even with CPU offloading

Decode speed is slower than expected

Model download is slow

MLX distributed inference

Next steps

Sources