Skip to main content
Quantization-aware training (QAT) produces low-bit weights that hold quality far better than a plain 4-bit conversion. Instead of shrinking a finished model in one shot, QAT simulates low precision during training, so the weights learn to tolerate rounding before they’re ever cast down. The payoff for running models locally: near-full-quality output at roughly a quarter of the VRAM. This page explains what QAT is, how it relates to the GGUF files you already download, and which QAT checkpoints you can run today. For the basics of quantization and how to read a name like Q4_K_M, start with Quantization explained.

​
QAT vs post-training quantization (PTQ)

Almost every quantized model you download is made with post-training quantization (PTQ): take a finished BF16 model, round its weights to fewer bits, ship it. It’s fast and needs no retraining — but the model was never given a chance to adapt to the rounding error, so accuracy loss grows as you push the bit-width down. QAT inserts “fake quantization” into the training or fine-tuning loop. The forward pass rounds and clamps weights to the target bit-width but keeps them in floating point, and gradients (via a straight-through estimator) teach the model to stay accurate under that rounding before it’s actually cast to low precision. Because the weights were optimized to survive quantization, a QAT checkpoint beats a PTQ one at the same bit-width. The gap is measurable:
  • PyTorch’s own numbers on Llama 3-8B at 4-bit weights: QAT recovered 96% of the accuracy degradation on HellaSwag and 68% of the perplexity degradation on WikiText versus PTQ. (PyTorch blog)
  • Google’s Gemma 3 QAT int4 checkpoints cut the perplexity gap versus the non-quantized model by 54% compared to a naive Q4_0 conversion of the same model. (Google Developers blog)
The recovery rate isn’t a fixed constant — it varies by model, bit-width, and benchmark. Treat the figures above as illustrative of the direction and rough magnitude, not a guarantee for every model.

​
QAT is a method, not a file format

A common mix-up is “QAT vs GGUF.” They aren’t alternatives — they answer different questions:
  • QAT and PTQ are how the low-bit weights were produced (the training/quantization method).
  • GGUF is the container the weights ship in for llama.cpp-based runtimes (Ollama, LM Studio, Jan).
A QAT checkpoint can still be exported to GGUF — and usually is. Google’s release makes this explicit in its naming: google/gemma-4-12B-it-qat-q4_0-gguf is a GGUF file whose Q4_0 weights came out of the QAT pipeline, as opposed to a plain gemma-4-12b-it that Ollama quantized to Q4_0 with ordinary PTQ. Same format, same bit-width — different, higher-fidelity weights inside. So the real comparison is QAT vs PTQ, and GGUF is just how either one reaches your runtime.

​
Gemma QAT: the checkpoints to actually run

Google is the standout here: it fine-tunes each Gemma model with QAT and publishes the results as ready-to-run int4 checkpoints. On Ollama they carry an -it-qat tag. For Gemma 4 (the current generation — see Gemma): Sizes are from ollama.com/library/gemma4 — confirm the current tag list there before you standardize on one. Google reports that Gemma 4 QAT yields “higher overall quality compared to standard PTQ baselines.” The well-quantified precedent is Gemma 3, where int4 QAT dropped VRAM roughly 3× — the 27B model from ~54 GB at BF16 to ~14 GB — while staying close to full-precision quality, which is what brought 27B-class models onto a single consumer GPU.
If a model offers both a QAT tag and a same-bit-width PTQ tag, prefer QAT — you get better quality at identical size and no extra cost at inference time. QAT pays its price once, during training.

​
Fine-tuning your own QAT model

QAT isn’t only for vendor checkpoints. Unsloth (with PyTorch’s TorchAO team) added QAT to its fine-tuning library: pass a qat_scheme (int4, int8-int4, fp8-int4, or fp8-fp8) so fake-quantization runs during your LoRA/fine-tune loop, then export to a low-bit format. Unsloth reports recovering up to ~70% of the accuracy a naive quantization loses, with no added inference-time overhead and the same on-disk footprint as ordinary quantization. This is the practical path for the popular open models that don’t ship official QAT checkpoints — Qwen, for example, has no first-party QAT release, but its models are a common base for community QAT recipes built on Unsloth and TorchAO. If you fine-tune and serve your own QAT model, serve your fine-tune covers exporting it to a runtime, and Tokios connects it from there.

​
Run your QAT model from anywhere

QAT gets a high-quality model to fit on the hardware you actually own. Once it’s running locally — on Ollama, LM Studio, vLLM, or llama.cpp — the next step is reaching it from your agents, apps, and teammates without exposing localhost.

Give it a stable endpoint

Tokios doesn’t pick, host, or fine-tune the model — once your QAT checkpoint runs locally, Tokios turns it into a stable, authenticated endpoint at https://api.tokios.com you can call from anywhere.

​
Go deeper

Quantization explained

How quantization works and how to read GGUF names like Q4_K_M.

Run Gemma locally

Gemma 4’s sizes, hardware fit, and its -it-qat quants.

Quantization and hardware

Which weight formats — GGUF, NVFP4, MXFP4, AWQ, MLX — run on your hardware.

What size model fits your GPU?

Check whether a model-and-quant combination fits before you download it.