Decode, prefill, and power across precision levels

Every local LLM deployment has 2 speeds that matter — and they respond differently to the precision knob. This page explains why dropping from FP16 to INT4 might triple your generation speed but barely move your prompt-processing time, and how to measure the trade-off on your own hardware.

Two speeds that matter

Decode throughput is how fast the model generates tokens after it starts answering, measured in tokens per second (tok/s). It determines how quickly a chat reply or code completion appears on screen. Prefill latency is how long the model takes to process your input before it generates the 1st token, measured in milliseconds per token (ms/token). It determines the pause between hitting Enter and seeing output start to stream. Different workloads care about different speeds:

Workload	What matters most	Why
Interactive chat	Decode throughput	Short prompts, long replies — generation dominates
Document analysis	Prefill latency	Long prompts (10K+ tokens), short replies — input processing dominates
Coding agents	Both	Moderate prompts, long multi-turn output — you feel both
Batch processing	Total throughput	Decode and prefill both contribute to wall-clock time

Time-to-first-token (TTFT) combines prefill latency with any scheduling or queue delay. For a single user on local hardware, TTFT is roughly equal to prefill time. Under concurrent load, queuing adds to TTFT independently.

What precision means for speed

Precision controls how many bits represent each weight in the model. Fewer bits means smaller weights, which changes 2 things: how much memory the model occupies and how much data the GPU reads per token.

Precision	Memory multiplier	Typical quality impact	Supported hardware
FP16 / BF16	1.0x (baseline)	None — full fidelity	All GPUs
FP8 (E4M3)	0.5x	<2% on modern models	Ada/Hopper/Blackwell (SM89+)
INT8 (W8A8)	0.5x	1-3% loss	Most GPUs with INT8 tensor cores
INT4 (W4A16, GPTQ, AWQ)	0.25x	2-5% loss	All GPUs (dequantized at runtime)
NVFP4	0.25x	1-3% loss	Blackwell / DGX Spark (SM121)

The quality column is approximate. Newer models trained with quantization-aware training (QAT) lose less quality at lower precisions. See QAT concepts for details.

How precision affects decode vs prefill differently

This is the key insight: decode and prefill hit different hardware bottlenecks, so the same precision change affects them unequally.

Decode is memory-bandwidth-bound

During decode, the model generates 1 token at a time. For each token, it reads the entire weight matrix from memory and performs a tiny amount of compute per weight. The GPU spends most of its time waiting for data to arrive from memory, not doing math. Smaller weights = less data to read = faster decode. The speedup is roughly proportional to the memory reduction:

FP16 → FP8: ~1.8-2.0x decode speedup
FP16 → INT4: ~2.5-3.5x decode speedup
FP16 → NVFP4: ~2.5-3.0x decode speedup

Prefill is compute-bound

During prefill, the model processes all input tokens in parallel. It reads the weights once but performs a large matrix multiplication across all prompt tokens. The GPU spends most of its time doing math, not waiting for memory. Quantization helps prefill less because the math is the bottleneck, not the weight reads:

FP16 → FP8: ~1.3-1.5x prefill speedup
FP16 → INT4: ~1.2-1.5x prefill speedup
FP16 → NVFP4: ~1.3-1.5x prefill speedup

Practical implication

As precision decreases, decode improves faster than prefill. At INT4, a model that felt sluggish at FP16 generates text rapidly but still pauses noticeably on long prompts. This asymmetry drives the workload-specific recommendations below.

Power draw across precision levels

Lower precision reduces power consumption for 2 reasons: fewer memory reads (memory access dominates energy cost) and simpler arithmetic (INT4 multiply costs less than FP16 multiply).

DGX Spark power envelope

DGX Spark’s GB10 chip has a ~40W TDP shared between Grace CPU and Blackwell GPU. Here are observed power ranges during inference:

Precision	Idle power	Decode load	Prefill load
FP16	8-12W	30-38W	35-40W
FP8	8-12W	25-33W	30-38W
NVFP4	8-12W	22-30W	28-35W

Consumer GPU power expectations

GPU	FP16 decode load	INT4/FP8 decode load	Savings
RTX 4090 (450W TDP)	300-400W	200-300W	~25% less at INT4
Strix Halo (120W TDP)	80-110W	60-85W	~20% less at INT4

Cost of 24/7 serving

For a DGX Spark running continuously at decode load:

Precision	Avg power	Monthly kWh (approx)
FP16	34W	~24 kWh
FP8	29W	~21 kWh
NVFP4	26W	~19 kWh

On a consumer GPU like the RTX 4090 at 10 hours/day:

Precision	Avg power	Monthly kWh (approx)
FP16	350W	~105 kWh
FP8	250W	~75 kWh
INT4	200W	~60 kWh

Benchmark methodology

Run these tests on your own setup to see how precision affects your specific model and hardware.

Measure decode throughput

Generate a fixed number of output tokens from a short prompt and measure tokens per second.

# Start the server first
vllm serve <model> --quantization <precision> --max-model-len 8192

# In another terminal
python -m vllm.benchmarks.benchmark_serving \
  --model <model> \
  --num-prompts 5 \
  --request-rate 1 \
  --max-input-len 128 \
  --max-output-len 256

./llama-bench \
  -m <model>.gguf \
  -p 512 \
  -n 128 \
  -ngl 99 \
  --repetitions 3

ollama run <model> --verbose "Write a short paragraph about inference speed."
# The --verbose flag prints eval rate in tok/s

Measure prefill latency

Time how long the model takes to produce its 1st token at varying prompt lengths.

# vLLM — vary --max-input-len to test prefill at different prompt sizes
python -m vllm.benchmarks.benchmark_serving \
  --model <model> \
  --num-prompts 3 \
  --request-rate 1 \
  --max-input-len 2048 \
  --max-output-len 1

Record time-to-first-token (TTFT) from the output. Repeat at 512, 1024, 4096, and 8192 input tokens.

Measure power draw

# Sample power every second during inference
watch -n 1 nvidia-smi --query-gpu=power.draw,power.limit --format=csv

# Reports SoC power on Grace/Blackwell
tegrastats --interval 1000

What to report

Fill in this table for each precision level you test:

Metric	FP16	FP8	INT4 / NVFP4
Decode (tok/s)
Prefill (ms/token, 4K prompt)
Prefill (ms/token, 16K prompt)
TTFT (s, 8K prompt)
Power at decode load (W)
Power idle (W)

Real-world decision table

Pick the row that matches your primary workload:

Your use case	Prioritize	Recommended precision	Why
Interactive chatbot	Decode speed	Q4_K_M or FP8	Short prompts, long replies — decode dominates your experience
Document analysis	Prefill speed	FP16 or FP8	Long prompts (10K+ tokens) — prefill dominates latency
Laptop / battery operation	Power efficiency	INT4 (Q4_K_M, AWQ)	Smallest memory footprint, lowest power draw
Multi-user serving	Both	FP8 with batching	Balance quality and throughput; FP8 KV cache doubles concurrency
Maximum output quality	Quality only	FP16 / BF16	Accept higher memory and power cost for zero quantization loss
DGX Spark deployment	Decode + memory fit	NVFP4	Native SM121 support, 75% size reduction, fits large models in 128GB

Why not always use INT4?

INT4 quantization (GPTQ, AWQ, Q4_K_M) reduces quality noticeably on smaller models and instruction-following tasks. On models under 13B parameters, the quality gap between FP16 and INT4 can degrade output. On 70B+ models, the gap narrows because the model has more capacity to absorb quantization noise. Test your specific model before committing.

Does KV cache precision affect these numbers?

Yes. FP8 KV cache (set with --kv-cache-dtype fp8 in vLLM) halves the KV cache memory independently of weight quantization. It has minimal impact on decode speed but lets you fit longer contexts or more concurrent users. See DGX Spark memory tuning for sizing tables.

How does context length change decode speed?

Decode reads the entire KV cache on every token. Longer conversations build larger caches, so decode slows down as context grows. A model at 4K context may generate at 40 tok/s but drop to 25 tok/s at 32K context on the same hardware. See KV cache and context for the mechanics.

Quick benchmark commands

Use these as starting points to compare precisions on a single model.

vLLM benchmark suite

# Launch with a specific quantization
vllm serve <model> --quantization fp8 --max-model-len 8192

# Run the built-in benchmark
python -m vllm.benchmarks.benchmark_serving \
  --model <model> \
  --num-prompts 10 \
  --request-rate 2 \
  --max-input-len 1024 \
  --max-output-len 128

# Repeat with --quantization nvfp4, then compare results

llama.cpp benchmark

# FP16 baseline
./llama-bench -m model-f16.gguf -p 512 -n 128 -ngl 99

# Q4_K_M quantized
./llama-bench -m model-q4_k_m.gguf -p 512 -n 128 -ngl 99

# Q8_0 for comparison
./llama-bench -m model-q8_0.gguf -p 512 -n 128 -ngl 99

llama.cpp prints a comparison table showing prompt processing (pp) and text generation (tg) speeds side by side.

Ollama built-in evaluation

# Run with verbose output to see eval speed
ollama run <model> --verbose "Summarize the theory of general relativity in 3 sentences."

# The output includes:
# eval count:   42 generation(s)
# eval duration: 1.8s
# eval rate:    23.3 tokens/s

Compare across precisions

Run the same prompt at each precision and record results in a table:

Precision	pp (ms/token)	tg (tok/s)	Memory (GB)
FP16
FP8
Q4_K_M

For the most reproducible results, warm up the model with 1 request before measuring, close other GPU workloads, and run at least 3 repetitions. Report the median value.

Next steps

DGX Spark memory tuning

KV cache sizing, batch trade-offs, and tuning parameters for 128GB unified memory.

vLLM on DGX Spark

Installation, NVFP4 configuration, and optimal launch commands.

Benchmark a deployment

End-to-end testing through Tokios — latency, streaming, and client comparison.

Quantization concepts

Deeper explanation of GPTQ, AWQ, GGUF, and quantization trade-offs.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Decode, prefill, and power across precision levels

Two speeds that matter

What precision means for speed

How precision affects decode vs prefill differently

Decode is memory-bandwidth-bound

Prefill is compute-bound

Practical implication

Power draw across precision levels

DGX Spark power envelope

Consumer GPU power expectations

Cost of 24/7 serving

Benchmark methodology

Measure decode throughput

Measure prefill latency

Measure power draw

What to report

Real-world decision table

Quick benchmark commands

vLLM benchmark suite

llama.cpp benchmark

Ollama built-in evaluation

Compare across precisions

Next steps

DGX Spark memory tuning

vLLM on DGX Spark

Benchmark a deployment

Quantization concepts

​Two speeds that matter

​What precision means for speed

​How precision affects decode vs prefill differently

​Decode is memory-bandwidth-bound

​Prefill is compute-bound

​Practical implication

​Power draw across precision levels

​DGX Spark power envelope

​Consumer GPU power expectations

​Cost of 24/7 serving

​Benchmark methodology

​Measure decode throughput

​Measure prefill latency

​Measure power draw

​What to report

​Real-world decision table

​Quick benchmark commands

​vLLM benchmark suite

​llama.cpp benchmark

​Ollama built-in evaluation

​Compare across precisions

​Next steps

DGX Spark memory tuning

vLLM on DGX Spark

Benchmark a deployment

Quantization concepts

Two speeds that matter

What precision means for speed

How precision affects decode vs prefill differently

Decode is memory-bandwidth-bound

Prefill is compute-bound

Practical implication

Power draw across precision levels

DGX Spark power envelope

Consumer GPU power expectations

Cost of 24/7 serving

Benchmark methodology

Measure decode throughput

Measure prefill latency

Measure power draw

What to report

Real-world decision table

Quick benchmark commands

vLLM benchmark suite

llama.cpp benchmark

Ollama built-in evaluation

Compare across precisions

Next steps