> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# FreeToken: MoE inference on consumer NVIDIA GPUs

> Run 35B-753B parameter MoE models on 8GB NVIDIA GPUs with FreeToken. UC Berkeley's edge-native engine uses dynamic CPU-GPU co-scheduling for 3-4x faster decode than Ollama.

FreeToken is an open-source inference engine from UC Berkeley that makes it practical to run frontier-scale Mixture-of-Experts (MoE) models on consumer NVIDIA GPUs. Where traditional serving stacks require you to fit the entire model in VRAM, FreeToken uses **dynamic CPU-GPU co-scheduling** to stream expert weights from system memory as needed — letting you run a 35B-parameter model on an 8GB GPU at \~39 tokens/s.

> [!NOTE]
>
> Last verified: 2026-10-20

## What FreeToken solves

MoE models like DeepSeek-V4-Flash (284B total, 8B active) or Qwen3.6-35B (30.5B total, 3.3B active) generate tokens at the speed of their **active parameters** — but they need memory for **all parameters**. A 35B MoE at 4-bit quantization needs \~18 GB of memory, which doesn't fit on an 8GB GPU.

Traditional offloading (llama.cpp, Ollama) moves weights to system RAM statically, which works but is slow — PCIe bandwidth becomes the bottleneck. FreeToken takes a different approach:

* **Dynamic co-scheduling** — the engine predicts which experts you'll need next and streams them from CPU to GPU ahead of time, overlapping transfer with compute
* **Double buffering** — while one set of experts runs on GPU, the next set transfers from CPU, hiding latency
* **Bandwidth-adaptive execution** — FreeToken measures your PCIe bandwidth and adjusts the scheduling policy to match

The result: 3-4x faster decode than static offloading on the same hardware.

## Supported models

FreeToken currently supports three MoE model families:

| Model | Total params | Active params | Memory at 4-bit | Status |
| - | - | - | - | - |
| **Qwen3.6-35B-A3B** | 30.5B | 3.3B | \~18 GB | Stable |
| **DeepSeek-V4-Flash** | 284B | 8B (in) / 16B (out) | \~170 GB | Stable |
| **GLM-5.2** | 753B | \~40B | \~452 GB | Experimental |

The 35B model fits on a single 24GB GPU at 4-bit. The 284B and 753B models require system RAM offloading — FreeToken's dynamic scheduling makes this practical on a gaming desktop with 64-128 GB of system RAM.

## Hardware requirements

**Minimum:**

* NVIDIA RTX 30/40/50 series GPU (8GB+ VRAM)
* 32 GB system RAM (64 GB+ recommended for 284B+ models)
* PCIe 3.0 x16 or better (PCIe 4.0/5.0 preferred)
* CUDA 12.x

**Not supported:**

* Apple Silicon (use MLX instead — see [MLX distributed inference](#mlx-distributed-inference))
* AMD GPUs
* Pre-Pascal NVIDIA GPUs (no CUDA compute capability 6.0+)

> [!TIP]
>
> FreeToken is optimized for NVIDIA's consumer GPU lineup. If you're on a Mac with Apple Silicon, the MLX framework with distributed inference across multiple Macs is the equivalent path for running 800B+ models locally.

## Installation

Install FreeToken with `uv` (recommended) or `pip`:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Using uv (faster, isolated environment)
uv pip install "freetoken[accel]"

# Using pip
pip install "freetoken[accel]"
```

The `[accel]` extra installs CUDA kernels and optimization dependencies. Without it, FreeToken falls back to a slower reference implementation.

## Quick start

### Serve Qwen3.6-35B on a 24GB GPU

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Download the model (4-bit quantization)
freetoken download qwen3.6-35b-a3b:q4

# Start the server
freetoken serve qwen3.6-35b-a3b:q4 --port 8000
```

FreeToken exposes an OpenAI-compatible API at `http://127.0.0.1:8000/v1`. Test it:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.6-35b-a3b:q4",
    "messages": [{"role": "user", "content": "Explain MoE routing in one paragraph."}],
    "max_tokens": 256
  }'
```

Expected performance on an RTX 4090 (24GB):

* **Decode speed:** \~39 tokens/s
* **Time to first token:** \~1.2s (4K prompt)
* **Memory usage:** \~18 GB VRAM + \~2 GB system RAM

### Serve DeepSeek-V4-Flash with CPU offloading

For the 284B model, you'll need 64+ GB of system RAM and a GPU with at least 8GB VRAM:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Download (this is ~170 GB at 4-bit)
freetoken download deepseek-v4-flash:q4

# Serve with CPU offloading enabled
freetoken serve deepseek-v4-flash:q4 \
  --port 8000 \
  --cpu-offload \
  --offload-ratio 0.8
```

The `--offload-ratio 0.8` tells FreeToken to keep 20% of experts in GPU memory and stream the remaining 80% from system RAM. Adjust based on your VRAM:

| GPU VRAM | Offload ratio | Expected decode speed |
| - | - | - |
| 8 GB | 0.9 | \~12 tokens/s |
| 12 GB | 0.85 | \~18 tokens/s |
| 16 GB | 0.75 | \~26 tokens/s |
| 24 GB | 0.6 | \~35 tokens/s |

## Connect to Tokios

Once FreeToken is running on `localhost:8000`, point the Tokios connector at it:

```json theme={"theme":{"light":"github-light","dark":"css-variables"}}
{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8000/v1"
  }
}
```

Register the model in the Tokios dashboard and you get an authenticated API endpoint you can call from anywhere. See [Supported backends](/setup/supported-backends.md) for the full connector setup flow.

## Performance tuning

### Optimize for throughput

If you're serving multiple concurrent requests, enable continuous batching:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
freetoken serve qwen3.6-35b-a3b:q4 \
  --port 8000 \
  --batching \
  --max-batch-size 8 \
  --max-wait-ms 10
```

This batches up to 8 requests together, amortizing the expert-loading cost across multiple users. Latency per request increases slightly, but total throughput improves.

### Optimize for latency

For interactive use where you want the fastest possible first token:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
freetoken serve qwen3.6-35b-a3b:q4 \
  --port 8000 \
  --prefill-priority high \
  --disable-batching
```

This prioritizes prefill (prompt processing) over decode, reducing time-to-first-token at the cost of lower overall throughput.

### Monitor resource usage

FreeToken includes a built-in metrics endpoint:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:8000/metrics
```

Returns:

* GPU memory usage
* System RAM usage
* PCIe transfer rate (GB/s)
* Decode throughput (tokens/s)
* Active expert count per token

Use this to tune your offload ratio — if PCIe transfer rate is low and decode speed is suffering, reduce the offload ratio to keep more experts in GPU memory.

## FreeToken vs other backends

| Backend | MoE support | 35B on 8GB GPU | Decode speed | Setup complexity |
| - | - | - | - | - |
| **FreeToken** | Native, optimized | Yes | \~39 tok/s (RTX 4090) | Medium (Python package) |
| **Ollama** | Static offloading | Yes | \~12 tok/s (RTX 4090) | Low (one command) |
| **llama.cpp** | Static offloading | Yes | \~15 tok/s (RTX 4090) | Medium (build from source) |
| **vLLM** | GPU-only | No (needs 24GB+) | \~45 tok/s (RTX 4090, full fit) | High (GPU server setup) |

FreeToken's advantage is the dynamic scheduling — it's 2.5-3x faster than static offloading on the same hardware when the model doesn't fully fit in VRAM. If your model **does** fit entirely in VRAM, vLLM is slightly faster, but FreeToken remains competitive.

## Troubleshooting

### "CUDA out of memory" even with CPU offloading

Your `--offload-ratio` is too low. Increase it to stream more experts from CPU:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
freetoken serve deepseek-v4-flash:q4 --cpu-offload --offload-ratio 0.95
```

### Decode speed is slower than expected

Check PCIe bandwidth:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
freetoken benchmark --model qwen3.6-35b-a3b:q4 --test pcie
```

If PCIe transfer rate is below 10 GB/s, you may have a bottleneck (older PCIe generation, or the GPU is in a slower slot). FreeToken's dynamic scheduling depends on fast CPU-GPU transfers.

### Model download is slow

FreeToken downloads from Hugging Face. If you're in a region with slow HF access, use a mirror:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
export HF_ENDPOINT=https://hf-mirror.com
freetoken download qwen3.6-35b-a3b:q4
```

## MLX distributed inference

If you're on Apple Silicon and want to run 800B+ models locally, MLX's distributed inference is the equivalent of FreeToken's CPU-GPU co-scheduling — but across multiple Macs.

MLX distributed (announced October 2026) uses RDMA over Thunderbolt to cluster Mac Studios or Mac Pros. A 4× M5 Ultra cluster achieves 36.73 tokens/s on Muse Spark Open (816B parameters).

Setup:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# On each Mac
pip install mlx-lm

# Start the cluster
mlx_lm.distributed \
  --model muse-spark-open:q4 \
  --nodes 4 \
  --parallelism tensor
```

See [Apple ML Research](https://machinelearning.apple.com/research/mlx-distributed-inference) for the full technical details.

## Next steps

> **[Supported backends](/setup/supported-backends.md)**
>
> Compare FreeToken to other local inference backends.

> **[MoE concepts](/concepts/moe-routing-and-expert-parallelism.md)**
>
> Understand how Mixture-of-Experts routing works and why memory scales with total parameters.

> **[Model fit](/guides/model-fit.md)**
>
> Calculate whether a model and quantization fit your hardware.

> **[Connect to Tokios](/quickstart.md)**
>
> Turn your local FreeToken server into an authenticated API endpoint.

## Sources

* [FreeToken GitHub repository](https://github.com/FlashML-org/FreeToken) — installation, supported models, and technical details.
* [FreeToken paper (arXiv)](https://arxiv.org/html/2608.16157v1) — bandwidth-adaptive execution and dynamic co-scheduling.
* [InfoQ coverage](https://www.infoq.com/news/2026/08/freetoken-local-inference/) — practical benchmarks and hardware requirements.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.