Skip to main content
Quantization trades precision for size and speed, and it has two independent axes: the quant name (Q4_K_M, INT4, FP8 — how many bits and how they’re packed, covered in Quantization explained) and the quant format (GGUF, AWQ, NVFP4 — the file format and kernel it runs through, which decides what hardware can actually load it). This page owns the second axis: pick a row below, and you know what hardware it needs.

​
The matrix

† vLLM’s own hardware-support table lists AWQ with x86/Arm CPU support via a separate code path, but not GPTQ; treat both as GPU-first formats in practice. ‡ vLLM’s compatibility table marks AWQ and GPTQ as unsupported on AMD GPU in vLLM specifically — this is a vLLM-runtime limitation, not a statement about every AWQ/GPTQ kernel everywhere. § Pre-Blackwell NVIDIA GPUs don’t have the Tensor Core support NVFP4 needs; it isn’t a driver-only limitation. ** MLX’s core PyPI package targets Apple Silicon + macOS; the project also ships an experimental Linux CUDA backend for the framework itself, but MLX-quantized model files you’ll find on Hugging Face are built for the Apple Metal path. Sources for every claim above are cited in the notes below.

​
Per-format notes

​
GGUF — the portable default

GGUF is llama.cpp’s model format, and it’s what Ollama and LM Studio use under the hood. It runs on CPU, Apple Metal, NVIDIA CUDA, AMD ROCm (via HIP), and Vulkan — llama.cpp’s own build docs list dedicated build targets for all of these backends, which makes GGUF the most portable format across consumer hardware. If you’re not sure what your hardware supports, GGUF is the safe default. See llama.cpp and the GGUF spec.

​
bf16 / FP16 — the baseline

bf16 and FP16 store each parameter in 2 bytes with no compression. This is the precision quantization compresses from — every Q4, AWQ, or NVFP4 file is a smaller, lossier version of a bf16/FP16 original. Treat support as framework-dependent: CUDA and ROCm serving stacks commonly support these precisions when you have enough memory, while CPU, Metal, and Vulkan paths depend on the runtime and are often impractical for large models. At roughly 4x the footprint of a 4-bit quantization, bf16/FP16 is rarely what you run locally on consumer hardware. See GPU-to-model fit for the memory math.

​
AWQ and GPTQ — GPU-oriented, vLLM’s formats

AWQ and GPTQ are 4-bit quantization formats built for GPU inference, and vLLM is the runtime that documents and maintains support for both. Per vLLM’s hardware-compatibility table, both run on NVIDIA GPUs (Volta through Hopper, with GPTQ additionally supported on Volta); vLLM’s table shows neither as supported on AMD GPU. Treat them as an NVIDIA/CUDA-first pair unless you’ve confirmed otherwise for your specific runtime. See vLLM’s quantization docs.

​
MXFP4 — gpt-oss’s native format

MXFP4 is the Open Compute Project’s Microscaling 4-bit floating-point format, and it’s the format OpenAI shipped gpt-oss in natively — both gpt-oss-120b and gpt-oss-20b are distributed pre-quantized in MXFP4 on Hugging Face, which is what lets 120b run in about 80GB and 20b in about 16GB. llama.cpp added native MXFP4 support for gpt-oss, and vLLM supports it as well. See OpenAI — Introducing gpt-oss and the llama.cpp gpt-oss support notes.

​
NVFP4 — Blackwell only

NVFP4 is NVIDIA’s own 4-bit floating-point format, purpose-built for the fifth-generation Tensor Cores in the Blackwell architecture (RTX 50 series consumer cards and Blackwell-generation datacenter GPUs). NVIDIA’s own technical blog describes NVFP4 as a Blackwell Tensor Core format using a two-level scaling scheme, and positions it alongside MXFP4 and FP4 as the three 4-bit formats Blackwell’s Tensor Cores support natively. If your GPU predates Blackwell, NVFP4 isn’t the format for you — look for a GGUF, AWQ, or GPTQ build of the same model instead. See NVIDIA — Introducing NVFP4.
Many NVFP4 repos on Hugging Face live under the nvidia/ org (for example ...-NVFP4 suffixed models). That naming is a strong hint you need Blackwell-generation hardware to get the native speedup — check the model card before you pull one on an older card.

​
MLX — Apple Silicon

MLX is Apple’s array framework, built around a unified-memory model where arrays live in shared memory across CPU and GPU. MLX-quantized models — the format LM Studio and other Mac-first tools offer alongside GGUF — target Apple Silicon on macOS 14 or later. The project’s own install docs list Apple Silicon + macOS as the primary supported path (with an experimental Linux/CUDA backend for the framework itself, separate from the model files you’ll find distributed in MLX format). If you’re not on a Mac, MLX isn’t an option — use GGUF instead. See MLX on GitHub and the MLX install docs.

​
Which should you pick?

Match the format to the hardware you already have, not the other way around:
  • NVIDIA, consumer card (RTX 40/30 series or older Blackwell-less cards) — GGUF for the easiest path (Ollama, LM Studio), or AWQ/GPTQ if you’re running vLLM and want the extra throughput of a GPU-native kernel.
  • NVIDIA Blackwell (RTX 50 series or Blackwell datacenter GPUs) — everything above still works, but look for an NVFP4 build if the model publishes one; it’s the format built for your Tensor Cores.
  • Apple Silicon — GGUF (via Metal) is the simplest and most common path; MLX is a Mac-native alternative that some tools (like LM Studio) offer side by side with GGUF.
  • AMD GPU — GGUF through ROCm or Vulkan. AWQ/GPTQ support on AMD is inconsistent across runtimes, so don’t plan around it.
  • CPU only — GGUF. It’s the only format in this matrix with real CPU inference support, and it’s what llama.cpp, Ollama, and LM Studio are built on.
For the quant name to pick within whichever format you land on (Q4_K_M vs Q8_0, for example), see Quantization explained. For sizing a specific model against your VRAM, see GPU-to-model fit.

Turn it into a stable endpoint

Once a quantized model runs locally — whatever format it’s in — Tokios doesn’t change how it runs. It gives you a stable, authenticated API endpoint to reach that same local deployment from anywhere.