Skip to main content
A Mixture of Experts (MoE) model looks like a huge model but only uses a small fraction of its weights on each token it generates. That separation — huge total capacity, small per-token compute — is why the latest generation of frontier open-weight models has moved to MoE. It is also the reason a model like DeepSeek-V3 with 671B total parameters only activates 37B per token, yet still needs enough memory to hold all 671B. This page explains how routing works, what expert parallelism means for serving, and how to size hardware for MoE models.
Last verified: 2026-10-14

​
What is Mixture of Experts

In a standard dense transformer, every token passes through every layer and activates every parameter on each forward pass. A 70B dense model uses all 70B parameters for every token. An MoE model replaces the feed-forward sublayer in selected transformer blocks with multiple parallel expert sub-networks and a router (also called a gating network) that decides which experts process each token. Most experts sit idle for any given token. A model with 256 experts might activate only 8 per token — roughly 3% of its total capacity per step. The result:
  • Total parameters determine how much memory the model needs (all experts must stay loaded).
  • Active parameters determine how much compute each token costs (only the selected experts run).
That is the core trade. MoE buys you the knowledge capacity of a very large model at the per-token compute cost of a much smaller one — but you still have to pay the memory bill for every expert, whether it fires or not.

​
How does MoE routing work

Each MoE layer contains a router that selects which experts process each token. The most common approach across production models is top-k gating:
  1. The router computes a score for every expert by multiplying the token’s hidden state by a learned weight matrix, then applies a softmax to produce a probability distribution across all experts.
  2. The top-k experts with the highest scores are selected. Their outputs are combined using the softmax-normalized weights as coefficients.
  3. All other experts produce no output for that token — they are skipped entirely.
The value of k varies by model. DeepSeek-V3 and Qwen3 use k=8. gpt-oss uses k=4. Kimi K3 uses 16 routed experts plus 2 shared experts per token.

​
Shared experts

Some architectures reserve a subset of shared experts that activate for every token, regardless of routing scores. Shared experts handle common patterns (language modeling basics, formatting) so the routed experts can specialize. Qwen3-Next uses 1 shared expert plus 10 routed experts. Kimi K3 uses 2 shared experts plus 16 routed experts. DeepSeek-V3 and the original Qwen3 MoE do not use shared experts — all active experts are selected by the router.

​
Load balancing

A router that always sends tokens to the same few experts defeats the purpose of MoE: those experts become bottlenecks while the rest sit idle. Production models use several strategies to keep expert usage balanced:
  • Auxiliary loss — a training-time penalty that pushes the router toward uniform expert usage. Most early MoE models (Switch Transformer, ST-MoE) use this approach. The cost is a slight reduction in specialization: the router optimizes for balance alongside quality.
  • Auxiliary-loss-free load balancing — DeepSeek-V3 introduced a bias-based approach that adjusts per-expert routing biases at inference time. If an expert is receiving more tokens than its target share, its bias decreases, making it less likely to be selected. This removes the quality tradeoff of auxiliary losses entirely.
  • Stable LatentMoE — Kimi K3 uses a learned latent routing mechanism that produces more stable expert assignments across tokens, reducing the variance that causes load imbalance in standard top-k gating.
The practical takeaway: load balancing is solved at training time, not at serving time. As a user running an MoE model locally, you do not configure routing — you configure memory and parallelism.

​
Why MoE memory scales with total parameters, not active

This is the single most important thing to understand before running an MoE model locally. All experts must be resident in memory at all times. The router decides dynamically — per token, per layer — which experts to activate. There is no way to know in advance which experts a given prompt will need, so every expert must be loaded and ready. Swapping experts from disk on demand is far too slow for interactive inference. That means the memory formula from Memory and offloading applies to total parameters, not active:
Active parameters only affect compute per token (and therefore generation speed), not memory.

​
Worked examples

The 1.2x factor accounts for KV cache, activations, and framework overhead at a modest context length — the same formula as dense models. See Memory and offloading for the full sizing derivation.

​
What this means in practice

A Qwen3-30B-A3B has 30.5B total parameters but only 3.3B active. It generates tokens at roughly the speed of a 3.3B dense model — but it needs memory for all 30.5B. At 4-bit, that is about 18 GB, which fits a 24 GB GPU. A dense 30B model at 4-bit would need about the same memory but generate tokens more slowly, because it processes all 30B parameters per token instead of 3.3B. Conversely, Kimi K3 has 2.8T total parameters and 104B active. It generates tokens at roughly the speed of a 104B dense model — but it needs about 1.4 TB of memory just for the weights at 4-bit. No consumer or prosumer hardware holds that.

​
MoE vs dense models

The key insight: MoE decouples knowledge capacity from per-token compute. A 671B-parameter MoE with 37B active can match or exceed a 70B dense model on quality while generating tokens at roughly 70B speed — but it needs 671B worth of memory.

​
Expert parallelism: serving MoE across multiple GPUs

When an MoE model does not fit on a single GPU, you need to distribute it. Expert parallelism is the MoE-specific strategy for doing this:
  • Each GPU holds a subset of the total experts. In a 256-expert model across 8 GPUs, each GPU holds 32 experts.
  • During the forward pass, the router’s decisions determine which tokens need to be sent to which GPU. Tokens routed to experts on GPU 3 are communicated to GPU 3 for processing, then the results come back.
  • Non-expert layers (attention, embeddings) are typically split using standard tensor parallelism, where each GPU holds a slice of every layer’s weight matrices.
Production serving runtimes like vLLM and TensorRT-LLM combine tensor parallelism for attention layers with expert parallelism for MoE layers. The result is called mixed parallelism. The vLLM documentation covers the configuration in detail; the key setting is that expert parallelism is a separate parallelism axis from tensor parallelism. The practical implication: MoE models benefit from high-bandwidth interconnects between GPUs (NVLink, InfiniBand) because tokens have to move between GPUs during the routing step. A cluster connected only by PCIe will see significant speed penalties from cross-GPU communication.

​
Which open-weight models use MoE

The current generation of frontier open-weight models has moved heavily toward MoE. Here is the landscape as of October 2026:

​
Sub-50B total (runnable on 1-2 consumer GPUs at 4-bit)

​
50B-250B total (multi-GPU or high-memory unified systems)

​
250B+ total (server-class hardware required)

Sources: model cards on HuggingFace and the official technical reports linked below.

​
Practical takeaway for local inference

MoE changes what the numbers on a model card mean:
  • Active parameters tell you how fast the model generates tokens. A 671B MoE with 37B active generates at roughly 37B speed.
  • Total parameters tell you how much memory the model needs. That same 671B MoE needs 671B worth of VRAM, not 37B.
  • Expert count and routing are architecture details that affect quality and training efficiency, but as a local inference user, you do not configure them — you configure memory and parallelism.
When you see an MoE model advertised as “2.4T parameters, 95B active,” read that as: “generates at 95B speed, needs 2.4T of memory.” Size your hardware for the total, enjoy the speed of the active count. Use the GPU model-fit guide to check whether a specific MoE model and quantization fit your hardware before downloading.

​
Go deeper

Memory and offloading

The full sizing formula, including bytes-per-parameter by precision and the offloading mechanics.

Quantization

Shrink bytes-per-parameter to change the memory equation for MoE and dense models alike.

How inference works

Why decode is memory-bandwidth-bound and how active parameters affect generation speed.

Model serving and scaling

Parallelism strategies, multi-GPU serving, and how runtimes distribute models.

​
Models using MoE

Kimi K3

2.8T total, 104B active, 896 experts with Stable LatentMoE routing.

DeepSeek

V3/R1 at 671B/37B, V4.1-Flash at 552B with asymmetric 8B/16B activation.

Qwen3

30B-A3B and 235B-A22B MoE tiers, 128 experts with top-8 routing.

Qwen3.8

2.4T flagship with 95B active, 512 experts, 10 routed + 1 shared.

GLM-5.3

753B total, ~40B active, MoE with 1M context.

gpt-oss

20b (32 experts) and 120b (128 experts), top-4 routing, Apache 2.0.

​
Once it fits and runs

Getting an MoE model loaded on your hardware is the first step. Once it is answering requests on localhost, the next step is reaching it from anywhere else.

Give it a stable endpoint

Tokios does not pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.

​
Frequently asked questions

The router selects different experts for each token, so every expert must be reachable. At any given moment only k experts fire, but the next token might need a completely different set. That means all expert weights must stay resident in memory (VRAM or RAM) at all times — you size by total parameters, not active parameters.
Calculate: total parameters × bytes per weight. At 4-bit (the native quantization floor for most MoE models), that is total_params × 0.5 GB. Add 20% for KV cache and overhead. If that number fits your combined VRAM + RAM, the model loads. If it does not fit, no amount of optimization changes that — you need more memory or a smaller model.
Shared experts are active on every token — they handle common tasks like language modeling. Routed experts are selected per token by the gating network. DeepSeek-V3 uses 256 routed experts with 8 active and no shared experts. Kimi K3 uses 896 experts with 16 routed + 2 shared. Qwen3.8 uses 512 experts with 10 routed + 1 shared.
Only the small ones. Qwen3-30B-A3B (30.5B total, ~18 GB at 4-bit) fits a 24 GB GPU. gpt-oss-20b (21B total, ~13 GB at 4-bit) also fits. Anything above ~50B total parameters needs either multi-GPU serving (vLLM, SGLang with expert parallelism) or a high-unified-memory system (Mac Studio, DGX Spark) with CPU offload.
Expert parallelism splits the expert layers across multiple GPUs — GPU 0 holds experts 0-63, GPU 1 holds experts 64-127, etc. Each GPU computes its experts’ outputs and they are combined via all-to-all communication. You need it when the total expert weights do not fit on a single GPU. vLLM and SGLang both support it; see the vLLM optimization guide for configuration.

​
Sources