Skip to main content
Quantization shrinks a model by storing its weights at lower numeric precision, trading a small amount of quality for a much smaller memory footprint and often faster inference. It’s the single biggest lever you have over whether a model fits on your hardware at all.

​
Why it works

A model’s weights are just numbers. Trained models are usually released at 16-bit precision (FP16 or BF16) — 2 bytes per parameter. Quantization rounds those numbers down to fewer bits: 8-bit uses 1 byte per parameter, and 4-bit (Q4) uses roughly half a byte. Fewer bits per weight means less data to store and less data to move through memory on every token generated, which is why quantized models are both smaller and often faster to run — recall from How inference works that decode speed is bound by how fast weights move through memory, not by raw compute. The tradeoff is precision loss. Rounding weights to fewer bits introduces small errors that compound across a model’s layers, which can measurably affect output quality at aggressive quantization levels. For exact bytes-per-parameter figures and the sizing math this feeds into, see Memory and offloading.

​
Reading GGUF quant names

Files distributed for llama.cpp-based backends (including Ollama and LM Studio) use the GGUF format, and quantized variants follow a naming convention documented in the llama.cpp quantize README and the GGUF format spec. A typical filename looks like Qwen3-8B-Instruct-Q4_K_M.gguf — the last segment before .gguf is the quantization. Quant names run roughly from Q2_K (smallest, lowest quality) up to Q8_0 (largest, near-lossless): So Q4_K_M decodes as: 4-bit, K-quant family, medium variant. GGUF Q4_K_M is mixed precision that averages out to roughly 0.5–0.56 bytes per parameter in practice — not a flat 4 bits across every tensor.
Higher number after Q is not strictly “always better” in every dimension — it’s a size/quality dial. Q8_0 is closer to the original 16-bit weights than Q2_K, but it’s also roughly four times the file size.

​
The size/quality tradeoff

Every step down in quantization saves memory and often gains a little speed, at some cost to output quality. The relationship isn’t linear — quality tends to hold up well down to around 4-bit, then degrades more noticeably below that:
  • Q8_0: Very close to full 16-bit quality. Use when you have VRAM to spare and want the safest quality margin.
  • Q4_K_M: The common default. Good balance of size and quality for most general use.
  • Q2_K–Q3_K: Smallest, most VRAM-friendly, but noticeably lower quality — reserve for cases where nothing else fits.

​
Which quant should you pick?

As a rule of thumb: start with Q4_K_M. It’s the default most model repos highlight for a reason — it fits comfortably on consumer hardware while staying close to full-precision output quality for most tasks. From there:
  • Go higher (Q5_K_M, Q6_K, Q8_0) if your VRAM has headroom after accounting for the KV cache at your target context length.
  • Go lower (Q3_K, Q2_K) only if Q4_K_M genuinely doesn’t fit and you have no other option — expect a noticeable quality drop, especially on reasoning-heavy or coding tasks.
Use the GPU model-fit guide to check whether a given model-and-quant combination fits your hardware before downloading it.
Quantization also affects other formats beyond GGUF — NVFP4, MXFP4, AWQ, GPTQ, and MLX all compress weights, but each is built for specific hardware and backends. See Quantization and hardware for which format runs where; this page covers GGUF naming and the general size/quality tradeoff only.

​
Quantization-aware training (QAT)

Everything above describes post-training quantization (PTQ) — quantizing a finished model in one shot. Quantization-aware training (QAT) instead simulates low precision during training, so the weights learn to tolerate rounding and hold quality better at the same bit-width. Google ships QAT checkpoints for Gemma under an -it-qat tag — for example ollama run gemma4:12b-it-qat — that stay close to full-precision quality at roughly a quarter of the BF16 VRAM. A QAT checkpoint still ships as GGUF: QAT is how the weights were made, not a different format.

Quantization-aware training, explained

How QAT differs from PTQ and GGUF, the Gemma QAT checkpoints to run, and fine-tuning your own with Unsloth.

​
Go deeper

Read a model card

Find which quant variants a model repo actually offers.

Memory and offloading

The full sizing formula, including bytes-per-parameter by precision.

Quantization and hardware

Match weight formats (GGUF, NVFP4, MXFP4, AWQ, GPTQ, MLX) to hardware.

​
Once you’ve picked a quant and it’s running

Choosing a quant gets a model to fit and run well on your hardware. Once it’s answering requests locally, the next step is reaching it from somewhere other than that one machine.

Give it a stable endpoint

Tokios doesn’t pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.