Tokens
The small chunks of text — roughly a few characters or a word-piece each — that a model actually reads and writes. Your prompt is split into tokens before the model sees it, and the reply is generated one token at a time. See How inference works.Prefill vs. decode
Prefill is the model reading your entire prompt at once, before writing anything back — it’s compute-bound and is what “time to first token” measures. Decode is the model generating the reply one token at a time — it’s memory-bandwidth-bound and is what tokens/sec measures. See How inference works.Dense vs. mixture-of-experts (MoE)
A dense model runs every parameter on every token. A mixture-of-experts model — named like30B-A3B, 30B total parameters with 3B active — only runs a subset of “expert” parameters per token, so it decodes faster than a dense model of the same total size, but still needs memory for the full parameter set. See Read a model card.
Precision
Precision is how many bits the runtime uses to store or compute with each number in the model, such asbf16, FP16, FP8, or 4-bit formats. Lower precision usually means less memory and faster data movement, but it can reduce quality if the quantization is too aggressive. See Quantization and Quantization formats and the hardware they need.
Inference framework
An inference framework is the serving engine that loads the model, runs the math, manages batching and the KV cache, and exposes an API. Ollama and LM Studio optimize for easy local use; vLLM, SGLang, llama.cpp, TensorRT-LLM, and MLX expose lower-level controls for throughput, parallelism, and hardware-specific kernels. See Serving and scaling.Kernel
A kernel is the low-level GPU or CPU routine that performs a hot operation such as matrix multiplication, attention, quantized dequantization, or sampling. Quantization formats are only useful when your runtime has kernels that understand them; for example,NVFP4 needs Blackwell Tensor Core support, while GGUF works broadly through llama.cpp backends. See Quantization formats and the hardware they need.
Tensor parallelism (TP)
Tensor parallelism splits one model layer’s weights and computation across multiple GPUs. Use it when the model is too large for one GPU or when one GPU cannot serve enough traffic. It reduces per-GPU memory pressure, but the GPUs must synchronize during each forward pass, so fast interconnects matter. vLLM exposes this as--tensor-parallel-size; TensorRT-LLM documents the same strategy as TP. See Serving and scaling.
Data parallelism (DP)
Data parallelism runs separate copies of the model on separate GPUs or instances so each copy can process different requests. It increases request throughput when the model already fits per replica. vLLM’s data-parallel deployment docs describe this as replicating model weights across instances or GPUs to process independent batches. See Serving and scaling.Expert parallelism (EP)
Expert parallelism applies to mixture-of-experts models. Instead of giving every GPU a full copy of every expert, the runtime places different experts on different GPUs and routes tokens to the GPU that owns the selected expert. This can improve locality and throughput for large MoE models, but it adds routing and synchronization overhead. See Serving and scaling.Multi-token prediction (MTP)
Multi-token prediction trains or adds auxiliary heads that predict more than one future token. At inference time, runtimes can use those extra predictions for speculative decoding: draft a few tokens, verify them with the main model, and keep the accepted tokens. Qwen’s Qwen3.6 model card shows MTP launch flags for vLLM and SGLang, and the DeepSeek-V3 technical report describes MTP as useful for speculative decoding. See Serving and scaling.Tool-calling
The ability for a model to return a structured request to call an external function or tool, rather than just plain text, so agentic clients can take actions. Reliable tool-calling is a major selection criterion for coding agents and other automated clients. See Which local model for which task?.Temperature and top-p
Sampling settings that control how predictable versus varied a model’s output is. Lower temperature makes output more deterministic and repetitive; higher temperature makes it more varied and occasionally less coherent. Top-p (nucleus sampling) restricts token choices to the smallest set whose combined probability reachesp, trimming away unlikely options before sampling. Both are request-level parameters your client sends alongside the prompt.
Embeddings
Numeric vector representations of text, used for search, retrieval, and similarity comparison rather than for generating conversational replies. Embedding models are a different tool from chat models — see Which local model for which task? for how they fit into model selection.Context window
The maximum number of tokens a model can hold in a single conversation at once, spanning both the prompt and the reply. A larger context window lets you send more text at once, but costs more memory through the KV cache. See KV cache and context.Quantization
Storing a model’s weights at lower numeric precision to shrink its size and often speed up inference, at some cost to output quality. GGUF quant names likeQ4_K_M encode the bit-width and quant family used. See Quantization.
KV cache
The running record of key/value representations a model builds while processing a conversation, so it doesn’t have to reprocess earlier tokens from scratch on every new token. It grows with context length and batch size, and sits in memory alongside the model weights. See KV cache and context.GGUF
Short for GGML Universal File — the file format used byllama.cpp-based backends (including Ollama and LM Studio) to store model weights, typically distributed as several quantized variants of the same model. It succeeds the project’s older GGML/GGJT weight formats, which is why you’ll still see “GGML” in older repos and tooling. See Read a model card and the GGUF format spec.
Now that the vocabulary is covered
These terms come up constantly once you start running models locally — bookmark this page and come back to it. Once a model is running and answering requests onlocalhost, the natural next step is reaching it from somewhere other than that one machine.
Give it a stable endpoint
Tokios doesn’t pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.