Step 1: tokenization
Before the model sees anything, your prompt is split into tokens — small chunks of text, roughly a few characters or a word-piece each. Tokenization is deterministic and near-instant; it’s just a lookup against the model’s vocabulary. The model never sees raw text, only a sequence of token ids.Step 2: prefill (compute-bound)
Prefill is the model reading your entire prompt — every token you sent, in one pass — before it writes a single word back. This step is compute-bound: the GPU is doing heavy matrix math across the whole prompt at once, so it scales with how much processing power you have. Prefill has two effects worth knowing:- It’s what you’re waiting on when a reply is slow to start. The delay between sending a prompt and seeing the first token is called time to first token (TTFT), and it’s almost entirely prefill time.
- It builds the KV cache — a running record of the prompt so the model doesn’t have to reprocess it on every subsequent token. See KV cache and context for what that cache costs in memory.
Step 3: decode (memory-bandwidth-bound)
Once prefill finishes, the model starts decode: generating the reply one token at a time. Each new token is appended to the sequence, and the next token is predicted from everything so far. Decode behaves very differently from prefill. On every single step, the model has to read its weights (or, for a mixture-of-experts model, just the active experts) plus the whole KV cache out of memory. Because so little new compute happens per token relative to the amount of data that has to move, decode is memory-bandwidth-bound — it’s limited by how fast data can travel from memory to the compute units, not by how much raw compute you have. This is what tokens per second measures, and it’s why:- A GPU with less compute but faster memory can out-decode a “faster” chip with slower memory.
- Longer context slows decode down, because there’s more KV cache to read on every single step, in addition to costing more memory (see KV cache and context).
Step 4: tokens stream back
As the model produces each token, it’s decoded back into text and streamed to the client immediately — this is why you see replies appear word by word instead of all at once. If you’re using Tokios, this stream travels from your local backend through the connector and gateway back to whatever app or agent sent the request; see How Tokios routes requests for that leg of the journey.What goes where in memory
Everything above happens against three things sitting in VRAM (or unified memory):
Weights are the floor — you can’t run the model without them fully loaded (or offloaded in part; see Memory and offloading). The KV cache is the part that grows as your conversation or document gets longer, which is why “it worked with a short prompt but ran out of memory on a long one” is such a common failure mode.
This page covers the mechanics. For the size math — how many gigabytes a given model and context actually need — see Memory and offloading and the GPU model-fit guide.
Go deeper
KV cache and context
Why longer context costs more memory and slows decode.
Memory and offloading
Sizing formulas and what happens when a model doesn’t fully fit.
Quantization
How shrinking weight precision changes both memory and speed.
Read a model card
Where to find a model’s architecture and context length before you run it.
Once you understand how it runs
Everything above happens on your machine, against a model running onlocalhost. Once you’re comfortable with that, the next question is how to reach it from somewhere other than your own machine.
Give your local model a stable endpoint
Tokios doesn’t pick, host, or fine-tune your model — once it’s running locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.