Skip to main content
Every local model needs enough memory to hold its weights, its KV cache, and some working overhead — and when it doesn’t fit entirely in VRAM, offloading decides what runs fast on the GPU versus slow on the CPU. This page covers the sizing theory and the offloading mechanics; for ready-made VRAM tables by GPU and model size, see the GPU model-fit guide — this page won’t repeat those tables.

​
The sizing formula

Model weights take a fixed number of bytes per parameter, set by precision: GGUF Q4_K_M is mixed precision and averages out to about 0.5–0.56 bytes per parameter rather than a flat 0.5, because K-quants assign different bit-widths to different tensors — see Quantization for how to read those names. Multiply parameter count by bytes-per-parameter to get raw weight size, then add roughly 20% on top for the KV cache, activations, and framework overhead at a modest context length:
Hugging Face measured this pattern directly on a 15B+ model: about 29 GB at bf16, ~15 GB at 8-bit, and ~9.5 GB at 4-bit — see the Hugging Face guide to optimizing LLMs. Longer context adds more KV cache on top of that 1.2× baseline; see KV cache and context for why that scales the way it does.
This formula is a planning estimate for weight size, not ground truth. The real number is the actual GGUF file size for the exact quantization you plan to run, plus the KV cache for your target context length. See the GPU model-fit guide for worked examples and an interactive estimator.

​
What’s actually competing for memory

Three things share the same pool of VRAM (or unified memory):
  • Model weights — fixed once the model is loaded, unaffected by prompt length.
  • KV cache — grows with context length and batch size; see KV cache and context.
  • Activations and framework overhead — smaller working memory needed during each forward pass, roughly what the 1.2× factor in the formula above accounts for.
Weights are the largest and most predictable cost. The KV cache is the part that can surprise you, because it’s invisible until your context grows.

​
Offloading: when a model doesn’t fully fit

If a model’s weights don’t fit entirely in your GPU’s VRAM, backends like llama.cpp let you split the model between GPU and CPU instead of failing outright. This is controlled with the --n-gpu-layers (short form -ngl) flag: it sets how many of the model’s layers get loaded onto the GPU, with the rest staying in system RAM.
  • Full offload (-ngl set high enough to cover every layer): the entire model runs on the GPU, at full GPU speed.
  • Partial offload: some layers run on the GPU, the rest run on the CPU using system RAM. This lets a model larger than your VRAM actually load and run, at a cost.
  • No offload (-ngl 0): the whole model runs on CPU only.
The cost of partial offload is speed, not correctness — the output is the same either way. CPU-side layers and system RAM have far lower memory bandwidth than GPU VRAM, and recall from How inference works that decode is memory-bandwidth-bound: every token generated has to read through the relevant layers, so any layer sitting in slower memory becomes a bottleneck on every single step. The more layers you push off the GPU, the slower generation gets — sometimes dramatically so once the model spills from RAM to disk.
If a model spills to disk (swap) rather than staying in RAM, expect a severe speed penalty — disk bandwidth is orders of magnitude slower than RAM, let alone VRAM. Treat disk spillover as a sign to pick a smaller model or a more aggressive quantization rather than something to run through.
Find the exact flag name and behavior for your backend in the llama.cpp repository; other backends (LM Studio, Ollama) expose the same idea through their own settings, usually built on llama.cpp underneath.

​
How unified memory changes the picture

On systems with unified memory — Apple Silicon and AMD’s Strix Halo platform — the CPU and GPU share a single memory pool instead of the GPU having its own separate, smaller VRAM. This changes the offloading calculus:
  • Capacity: a unified-memory system can load a much larger model than a discrete GPU with fixed VRAM, because the “VRAM” pool can be as large as the machine’s total RAM (up to the portion assigned to graphics use).
  • Bandwidth: that same pool runs at lower bandwidth than dedicated GDDR memory on a discrete GPU. A model that fits entirely in unified memory still decodes more slowly than the same model fitting entirely in a discrete GPU’s VRAM, because decode speed tracks bandwidth, not just capacity.
In practice, unified memory shifts the tradeoff from “does it fit at all” to “how fast does it run once it fits” — it solves the capacity problem that forces offloading in the first place, but doesn’t give you GDDR-class bandwidth in exchange. See the GPU model-fit guide for how specific unified-memory systems compare to discrete GPUs.

​
Go deeper

How inference works

Why decode is memory-bandwidth-bound in the first place.

KV cache and context

The part of memory usage that grows with your conversation, not just your model.

Quantization

Shrink bytes-per-parameter to change the sizing formula’s biggest input.

GPU model-fit guide

VRAM tables, an interactive estimator, and what fits on common hardware.

​
Once it fits and runs

Sizing and offloading get a model running on your hardware, whether that’s fully on GPU or split with the CPU. Once it’s answering requests on localhost, the next step is reaching it from anywhere else.

Give it a stable endpoint

Tokios doesn’t pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.