Skip to main content
When you run ollama run qwen3.6, a lot is hidden from you. The moment you open a model page’s advanced Run it tab and see a line like vllm serve … --tensor-parallel-size 8 --enable-expert-parallel, you meet a new vocabulary. This page explains those words — what each one does and when you’d actually need it.
If you run one model on one GPU (or a Mac) through Ollama or LM Studio, you can skip most of this. Parallelism and kernels matter when a model is too big for one GPU, or when you’re serving many users at once. Read this when a model page’s advanced tab sends you here.

​
The inference framework

An inference framework is the engine that loads the model, runs the math, manages batching and the KV cache, and exposes an API. They trade ease of use for control: The easy ones (Ollama, LM Studio) are the beginner path on every model page. The rest expose the controls below. Tokios connects to any of them — it forwards to whatever OpenAI-compatible /v1 your framework exposes.

​
Precision

Precision is how many bits the runtime uses to store and compute each number in the model — FP32 (32-bit), FP16/bf16 (16-bit), FP8 (8-bit), or 4-bit formats. Fewer bits means less memory and faster data movement, at some risk to quality. This is the same idea as quantization — precision is the knob, quantization is turning it down. Which low-precision format your hardware can actually run is a separate question, answered in Quantization formats and the hardware they need.

​
Kernels

A kernel is the small, highly optimized GPU (or CPU) routine that performs one hot operation — a matrix multiply, an attention step, dequantizing a 4-bit weight, sampling the next token. You rarely name kernels directly, but they explain two things you do see:
  • Why a quant format needs specific hardware. A format is only fast if the runtime ships a kernel that understands it on your chip. NVFP4 needs Blackwell Tensor Core kernels; GGUF runs broadly because llama.cpp ships kernels for CPU, Metal, CUDA, ROCm, and Vulkan. See Quantization formats and the hardware they need.
  • Why “FlashAttention” comes up. FlashAttention is a faster attention kernel that saves memory on long context. When a runtime says it’s enabled, that’s a kernel choice working for you.

​
Running across multiple GPUs — parallelism

When a model is too big for one GPU, or one GPU can’t serve enough traffic, the runtime spreads the work across several. There are four common strategies, and they answer different problems:
  • Tensor parallelism cuts each layer’s matrices into slices, one per GPU, and the GPUs combine results every forward pass. It lowers per-GPU memory, but the GPUs must talk constantly — fast interconnects (NVLink) matter.
  • Pipeline parallelism puts different layers on different GPUs, like a factory line. It scales across many devices but can leave GPUs idle waiting for the stage before them.
  • Data parallelism just runs independent copies. It adds no help for a model that already doesn’t fit — it’s purely for handling more simultaneous requests.
  • Expert parallelism applies only to MoE models: instead of copying every expert to every GPU, each GPU holds some experts and tokens are routed to the GPU that owns the chosen expert. Great for big MoE models, at the cost of routing overhead.
Sources: vLLM parallelism and scaling, vLLM data-parallel deployment, vLLM expert-parallel deployment, TensorRT-LLM parallelism strategy.

​
Speculative decoding and MTP

Decode is slow because the model reads its weights from memory for every single token. Speculative decoding speeds it up without changing the output: a fast “draft” proposes several tokens at once, then the full model verifies them in one pass and keeps the ones it agrees with. Because verification is cheaper than generating from scratch, you get more accepted tokens per memory read. Multi-token prediction (MTP) is one way to produce those drafts. The model ships extra “heads” trained to predict the next few tokens, so it can draft and verify itself. This is what the -mtp tags on Ollama (for example qwen3.6:35b-a3b-mtp-q4_K_M) enable. Qwen3.6’s model card documents MTP launch flags for vLLM and SGLang, and the DeepSeek-V3 technical report describes MTP for speculative decoding. Sources: Qwen3.6-27B model card, DeepSeek-V3 technical report, vLLM speculative decoding.
On a single consumer GPU, the win you can grab today is an -mtp tag if your runtime supports it — it can speed up generation with no quality loss. The parallelism strategies above only pay off once you have multiple GPUs.

​
Where this fits

How inference works

The prefill/decode basics these controls optimize.

Quantization & hardware

Which precision format runs on your chip.

What size model fits your GPU?

Size a model before you reach for multiple GPUs.

Give it a Tokios endpoint

However you serve it, Tokios turns your local /v1 into a stable, authenticated endpoint you can reach from anywhere.