> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Mixture of Experts (MoE): routing, expert parallelism, and memory

> Understand Mixture of Experts (MoE): how routing works, why memory scales with total params, and which open-weight models use MoE. Includes sizing formulas and hardware guide.

A **Mixture of Experts (MoE)** model looks like a huge model but only uses a small fraction of its weights on each token it generates. That separation — huge total capacity, small per-token compute — is why the latest generation of frontier open-weight models has moved to MoE. It is also the reason a model like DeepSeek-V3 with 671B total parameters only activates 37B per token, yet still needs enough memory to hold all 671B. This page explains how routing works, what expert parallelism means for serving, and how to size hardware for MoE models.

> [!NOTE]
>
> Last verified: 2026-10-14

## What is Mixture of Experts

In a standard **dense** transformer, every token passes through every layer and activates every parameter on each forward pass. A 70B dense model uses all 70B parameters for every token.

An MoE model replaces the feed-forward sublayer in selected transformer blocks with multiple parallel **expert** sub-networks and a **router** (also called a **gating network**) that decides which experts process each token. Most experts sit idle for any given token. A model with 256 experts might activate only 8 per token — roughly 3% of its total capacity per step.

The result:

* **Total parameters** determine how much memory the model needs (all experts must stay loaded).
* **Active parameters** determine how much compute each token costs (only the selected experts run).

That is the core trade. MoE buys you the knowledge capacity of a very large model at the per-token compute cost of a much smaller one — but you still have to pay the memory bill for every expert, whether it fires or not.

## How does MoE routing work

Each MoE layer contains a router that selects which experts process each token. The most common approach across production models is **top-k gating**:

1. The router computes a score for every expert by multiplying the token's hidden state by a learned weight matrix, then applies a softmax to produce a probability distribution across all experts.
2. The top-k experts with the highest scores are selected. Their outputs are combined using the softmax-normalized weights as coefficients.
3. All other experts produce no output for that token — they are skipped entirely.

The value of k varies by model. DeepSeek-V3 and Qwen3 use k=8. gpt-oss uses k=4. Kimi K3 uses 16 routed experts plus 2 shared experts per token.

### Shared experts

Some architectures reserve a subset of **shared experts** that activate for every token, regardless of routing scores. Shared experts handle common patterns (language modeling basics, formatting) so the routed experts can specialize. Qwen3-Next uses 1 shared expert plus 10 routed experts. Kimi K3 uses 2 shared experts plus 16 routed experts. DeepSeek-V3 and the original Qwen3 MoE do not use shared experts — all active experts are selected by the router.

### Load balancing

A router that always sends tokens to the same few experts defeats the purpose of MoE: those experts become bottlenecks while the rest sit idle. Production models use several strategies to keep expert usage balanced:

* **Auxiliary loss** — a training-time penalty that pushes the router toward uniform expert usage. Most early MoE models (Switch Transformer, ST-MoE) use this approach. The cost is a slight reduction in specialization: the router optimizes for balance alongside quality.
* **Auxiliary-loss-free load balancing** — DeepSeek-V3 introduced a bias-based approach that adjusts per-expert routing biases at inference time. If an expert is receiving more tokens than its target share, its bias decreases, making it less likely to be selected. This removes the quality tradeoff of auxiliary losses entirely.
* **Stable LatentMoE** — Kimi K3 uses a learned latent routing mechanism that produces more stable expert assignments across tokens, reducing the variance that causes load imbalance in standard top-k gating.

The practical takeaway: load balancing is solved at training time, not at serving time. As a user running an MoE model locally, you do not configure routing — you configure memory and parallelism.

## Why MoE memory scales with total parameters, not active

This is the single most important thing to understand before running an MoE model locally.

**All experts must be resident in memory at all times.** The router decides dynamically — per token, per layer — which experts to activate. There is no way to know in advance which experts a given prompt will need, so every expert must be loaded and ready. Swapping experts from disk on demand is far too slow for interactive inference.

That means the memory formula from [Memory and offloading](/concepts/memory-and-offloading.md) applies to **total parameters**, not active:

```text theme={"theme":{"light":"github-light","dark":"css-variables"}}
VRAM(GB) ≈ total_params(B) × bytes-per-param × 1.2
```

Active parameters only affect compute per token (and therefore generation speed), not memory.

### Worked examples

| Model | Total params | Active params | Experts (active/total) | Memory at 4-bit (\~0.5 B/param) | Memory at 8-bit (\~1 B/param) |
| - | - | - | - | - | - |
| DeepSeek-V3 / R1 | 671B | 37B | 8 of 256 | \~403 GB | \~805 GB |
| Kimi K2 | 1T | 32B | 8 of 384 | \~600 GB | \~1,200 GB |
| Kimi K3 | 2.8T | 104B | 16+2 of 896 | \~1,680 GB | \~3,360 GB |
| Qwen3-30B-A3B | 30.5B | 3.3B | 8 of 128 | \~18 GB | \~37 GB |
| Qwen3-235B-A22B | 235B | 22B | 8 of 128 | \~141 GB | \~282 GB |
| Qwen3.8-2.4T-A95B | 2.4T | 95B | 10+1 of 512 | \~1,440 GB | \~2,880 GB |
| GLM-5.3 | 753B | \~40B | Not publicly specified | \~452 GB | \~904 GB |
| gpt-oss-20b | 21B | 3.6B | 4 of 32 | \~13 GB | \~25 GB |
| gpt-oss-120b | 117B | 5.1B | 4 of 128 | \~70 GB | \~140 GB |

The 1.2x factor accounts for KV cache, activations, and framework overhead at a modest context length — the same formula as dense models. See [Memory and offloading](/concepts/memory-and-offloading.md) for the full sizing derivation.

### What this means in practice

A Qwen3-30B-A3B has 30.5B total parameters but only 3.3B active. It generates tokens at roughly the speed of a 3.3B dense model — but it needs memory for all 30.5B. At 4-bit, that is about 18 GB, which fits a 24 GB GPU. A dense 30B model at 4-bit would need about the same memory but generate tokens more slowly, because it processes all 30B parameters per token instead of 3.3B.

Conversely, Kimi K3 has 2.8T total parameters and 104B active. It generates tokens at roughly the speed of a 104B dense model — but it needs about 1.4 TB of memory just for the weights at 4-bit. No consumer or prosumer hardware holds that.

## MoE vs dense models

| | Dense | MoE |
| - | - | - |
| **Parameters per token** | All parameters | Only active parameters (a fraction of total) |
| **Memory** | Scales with total parameters | Scales with total parameters (same formula) |
| **Compute per token** | Scales with total parameters | Scales with active parameters only |
| **Generation speed** | Limited by total parameter count | Limited by active parameter count — often much faster than a dense model of the same total size |
| **Knowledge capacity** | Limited by total parameters | Total parameters can be much larger for the same per-token compute |
| **Hardware requirement** | Must fit total parameters | Must fit total parameters (not active) |
| **Multi-GPU serving** | Tensor parallelism splits layers | Expert parallelism splits experts across GPUs |

The key insight: MoE decouples knowledge capacity from per-token compute. A 671B-parameter MoE with 37B active can match or exceed a 70B dense model on quality while generating tokens at roughly 70B speed — but it needs 671B worth of memory.

## Expert parallelism: serving MoE across multiple GPUs

When an MoE model does not fit on a single GPU, you need to distribute it. **Expert parallelism** is the MoE-specific strategy for doing this:

* Each GPU holds a subset of the total experts. In a 256-expert model across 8 GPUs, each GPU holds 32 experts.
* During the forward pass, the router's decisions determine which tokens need to be sent to which GPU. Tokens routed to experts on GPU 3 are communicated to GPU 3 for processing, then the results come back.
* Non-expert layers (attention, embeddings) are typically split using standard **tensor parallelism**, where each GPU holds a slice of every layer's weight matrices.

Production serving runtimes like vLLM and TensorRT-LLM combine tensor parallelism for attention layers with expert parallelism for MoE layers. The result is called **mixed parallelism**. The vLLM documentation covers the configuration in detail; the key setting is that expert parallelism is a separate parallelism axis from tensor parallelism.

The practical implication: MoE models benefit from high-bandwidth interconnects between GPUs (NVLink, InfiniBand) because tokens have to move between GPUs during the routing step. A cluster connected only by PCIe will see significant speed penalties from cross-GPU communication.

## Which open-weight models use MoE

The current generation of frontier open-weight models has moved heavily toward MoE. Here is the landscape as of October 2026:

### Sub-50B total (runnable on 1-2 consumer GPUs at 4-bit)

| Model | Total | Active | Experts | Active per token | License |
| - | - | - | - | - | - |
| gpt-oss-20b | 21B | 3.6B | 32 | 4 | Apache 2.0 |
| Qwen3-30B-A3B | 30.5B | 3.3B | 128 | 8 | Apache 2.0 |

### 50B-250B total (multi-GPU or high-memory unified systems)

| Model | Total | Active | Experts | Active per token | License |
| - | - | - | - | - | - |
| gpt-oss-120b | 117B | 5.1B | 128 | 4 | Apache 2.0 |
| Qwen3-235B-A22B | 235B | 22B | 128 | 8 | Apache 2.0 |

### 250B+ total (server-class hardware required)

| Model | Total | Active | Experts | Active per token | License |
| - | - | - | - | - | - |
| DeepSeek-V3 / R1 | 671B | 37B | 256 | 8 | MIT (V3), custom (R1) |
| GLM-5.3 | 753B | \~40B | MoE | MoE | Custom (see [GLM-5.3](/models/glm-5.3.md)) |
| Kimi K2 | 1T | 32B | 384 | 8 | Custom |
| Qwen3.8-2.4T-A95B | 2.4T | 95B | 512 | 10+1 shared | Apache 2.0 |
| Kimi K3 | 2.8T | 104B | 896 | 16+2 shared | Custom |
| DeepSeek V4.1-Flash | 552B | 8B in / 16B out | MoE | Asymmetric | MIT |

Sources: model cards on HuggingFace and the official technical reports linked below.

## Practical takeaway for local inference

MoE changes what the numbers on a model card mean:

* **Active parameters** tell you how fast the model generates tokens. A 671B MoE with 37B active generates at roughly 37B speed.
* **Total parameters** tell you how much memory the model needs. That same 671B MoE needs 671B worth of VRAM, not 37B.
* **Expert count and routing** are architecture details that affect quality and training efficiency, but as a local inference user, you do not configure them — you configure memory and parallelism.

When you see an MoE model advertised as "2.4T parameters, 95B active," read that as: "generates at 95B speed, needs 2.4T of memory." Size your hardware for the total, enjoy the speed of the active count.

Use the [GPU model-fit guide](/guides/model-fit.md) to check whether a specific MoE model and quantization fit your hardware before downloading.

## Go deeper

> **[Memory and offloading](/concepts/memory-and-offloading.md)**
>
> The full sizing formula, including bytes-per-parameter by precision and the offloading mechanics.

> **[Quantization](/concepts/quantization.md)**
>
> Shrink bytes-per-parameter to change the memory equation for MoE and dense models alike.

> **[How inference works](/concepts/how-inference-works.md)**
>
> Why decode is memory-bandwidth-bound and how active parameters affect generation speed.

> **[Model serving and scaling](/concepts/model-serving-and-scaling.md)**
>
> Parallelism strategies, multi-GPU serving, and how runtimes distribute models.

## Models using MoE

> **[Kimi K3](/models/kimi-k3.md)**
>
> 2.8T total, 104B active, 896 experts with Stable LatentMoE routing.

> **[DeepSeek](/models/deepseek.md)**
>
> V3/R1 at 671B/37B, V4.1-Flash at 552B with asymmetric 8B/16B activation.

> **[Qwen3](/models/qwen3.6.md)**
>
> 30B-A3B and 235B-A22B MoE tiers, 128 experts with top-8 routing.

> **[Qwen3.8](/models/qwen3.8.md)**
>
> 2.4T flagship with 95B active, 512 experts, 10 routed + 1 shared.

> **[GLM-5.3](/models/glm-5.3.md)**
>
> 753B total, \~40B active, MoE with 1M context.

> **[gpt-oss](/models/gpt-oss)**
>
> 20b (32 experts) and 120b (128 experts), top-4 routing, Apache 2.0.

## Once it fits and runs

Getting an MoE model loaded on your hardware is the first step. Once it is answering requests on `localhost`, the next step is reaching it from anywhere else.

> **[Give it a stable endpoint](/quickstart.md)**
>
> Tokios does not pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.

## Frequently asked questions

> **Why does an MoE model need memory for all experts if only a few are active?**
>
> The router selects different experts for each token, so every expert must be reachable. At any given moment only k experts fire, but the next token might need a completely different set. That means all expert weights must stay resident in memory (VRAM or RAM) at all times — you size by total parameters, not active parameters.

> **How do I know if my hardware can run an MoE model?**
>
> Calculate: total parameters × bytes per weight. At 4-bit (the native quantization floor for most MoE models), that is `total_params × 0.5 GB`. Add 20% for KV cache and overhead. If that number fits your combined VRAM + RAM, the model loads. If it does not fit, no amount of optimization changes that — you need more memory or a smaller model.

> **What is the difference between shared experts and routed experts?**
>
> Shared experts are active on every token — they handle common tasks like language modeling. Routed experts are selected per token by the gating network. DeepSeek-V3 uses 256 routed experts with 8 active and no shared experts. Kimi K3 uses 896 experts with 16 routed + 2 shared. Qwen3.8 uses 512 experts with 10 routed + 1 shared.

> **Can I run an MoE model on a single GPU?**
>
> Only the small ones. Qwen3-30B-A3B (30.5B total, \~18 GB at 4-bit) fits a 24 GB GPU. gpt-oss-20b (21B total, \~13 GB at 4-bit) also fits. Anything above \~50B total parameters needs either multi-GPU serving (vLLM, SGLang with expert parallelism) or a high-unified-memory system (Mac Studio, DGX Spark) with CPU offload.

> **What is expert parallelism and when do I need it?**
>
> Expert parallelism splits the expert layers across multiple GPUs — GPU 0 holds experts 0-63, GPU 1 holds experts 64-127, etc. Each GPU computes its experts' outputs and they are combined via all-to-all communication. You need it when the total expert weights do not fit on a single GPU. vLLM and SGLang both support it; see the [vLLM optimization guide](/guides/vllm-optimization.md) for configuration.

## Sources

* [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437) — 671B total, 37B active, 256 experts, auxiliary-loss-free load balancing.
* [Kimi K2: Open Agentic Intelligence](https://arxiv.org/abs/2507.20534) — 1T total, 32B active, 384 experts, 8 active per token.
* [Kimi K3 model card](https://huggingface.co/moonshotai/Kimi-K3) — 2.8T total, 104B active, 896 experts, Stable LatentMoE.
* [Qwen3 Technical Report](https://arxiv.org/abs/2505.09388) — 128 experts, 8 active, no shared experts in Qwen3 MoE.
* [Qwen3-Next-80B-A3B](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd) — 512 experts, 10 routed + 1 shared.
* [Qwen3.8-Max announcement](https://qwen.ai/blog?id=qwen3.8) — 2.4T total, 95B active, 512 experts.
* [gpt-oss model card](https://arxiv.org/abs/2508.10925) — 32 experts (20b) and 128 experts (120b), 4 active per token.
* [GLM-5.3 model card](https://huggingface.co/zai-org/GLM-5.3) — 753B total, \~40B active.
* [Mixture of Experts Explained (localaimaster.com)](https://localaimaster.com/blog/mixture-of-experts-explained) — Routing mechanics and memory implications.
* [Mixture-of-Experts LLMs (Cameron R. Wolfe)](https://cameronrwolfe.substack.com/p/moe-llms) — Top-k gating formulation and training dynamics.
* [MoE training in Qwen3-235B and Kimi-K2](https://medium.com/clickbait-programming/moe-training-in-qwen3-235b-a22b-and-kimi-k2-c51f582d2ef5) — Learned routing comparison.
* [vLLM MoE Playbook](https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html) — Expert parallelism and mixed parallelism configuration.
* [Expert Parallelism in TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/advanced/expert-parallelism.html) — Multi-GPU expert distribution strategies.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.