Interactive model-fit finder — select hardware or model, adjust quantization and context.
How it works
The widget estimates memory requirements using:
MoE models (like Qwen3-235B-A22B or DeepSeek-V3) load all expert weights into memory, so total parameter count determines memory usage — even though only the active subset runs per token.
The sizing formula
Model weights take a fixed number of bytes per parameter, and that byte count depends entirely on precision:
Raw weight size is not the whole story. Add roughly 20% overhead (
×1.2) for the KV cache, activations, and framework memory at a modest context length — longer contexts add more KV cache on top. That gives the working formula:
~Q4 (4-bit), including the ~1.2 overhead:
For reference, higher precision scales the same models far past what a single consumer card can hold:
70B at FP16 is about 140 GB, and 120B at FP16 is about 230 GB.
Hugging Face measured a similar pattern on a 15B+ model: about 29 GB at bf16, ~15 GB at 8-bit, and ~9.5 GB at 4-bit. See the Hugging Face guide to optimizing LLMs and the llama.cpp quantize README for the underlying numbers — GGUF Q4_K_M uses mixed precision that averages out to about 0.5–0.56 bytes per parameter.
These formulas estimate weight size. The actual number that matters is the
GGUF file size listed on Hugging Face for the exact quantization you plan to run, plus the KV cache for your target context length. Always check that before committing to hardware.GPU VRAM at a glance
Sources: NVIDIA GeForce RTX 50 series, NVIDIA professional GPUs, Wikipedia: GeForce RTX 40 series, Wikipedia: GeForce RTX 30 series, AMD Ryzen AI Max, Apple technical specifications.
Dedicated GDDR memory (the NVIDIA cards above) is fast and fixed in size — you cannot add more later. Unified memory (Apple Silicon, Strix Halo) shares one pool between CPU and GPU, so it can be much larger, but it runs at lower bandwidth than a dedicated card.
What fits where
The table below estimates comfort level at~Q4 quantization for a modest context length. Treat every entry as an estimate — verify against the actual GGUF file size plus KV cache before you commit.
Ground truth is always the
GGUF file size on Hugging Face for the quantization you choose, plus the KV cache your context length requires.
Why unified memory loads big models but runs them slower
Loading a model only requires enough memory capacity to hold the weights. Running it fast requires enough memory bandwidth to keep feeding the compute units, because single-stream decoding is memory-bandwidth-bound — each token generated has to read the model’s weights (or, for a mixture-of-experts model, the active experts) out of memory. Prefill (processing the prompt) is compute-bound instead, so it behaves differently. This is why the two memory types trade off differently:- Dedicated GDDR (RTX 4090, 5090, and similar): high bandwidth (roughly
1000–1800 GB/s), but capacity is fixed by the card. - Unified memory (Apple Silicon, Strix Halo): much larger capacity (up to
128 GB), but lower bandwidth (roughly256–546 GB/s).
96 GB unified-memory machine can load a 120B model that would never fit on any single consumer GPU. It will still generate tokens more slowly than a dedicated GPU running a model that fits within the GPU’s own VRAM, because it has less bandwidth to move weights through per token.
Do not rely on published tok/s numbers from other setups — hardware, driver versions, quantization, context length, and batch size all shift the result. Measure your own deployment with llama-bench and see Benchmark your deployment for a repeatable way to compare deployments once your model is running.
Next steps
Get started with Tokios
Once a model fits your hardware, Tokios turns it into a stable API endpoint you can call from anywhere.
Pick the right model for your task
Sizing narrows down what fits. Task fit narrows down which family to load in the first place.