Optimize vLLM for throughput and latency

vLLM is the most feature-rich open-source runtime for serving large language models on GPUs. It ships with a wide range of configuration options that directly affect throughput, latency, and memory usage. This guide covers the knobs that matter most and how to set them for your workload.

Last verified: 2026-10-14 — vLLM v0.31.0.

V1 is the default. The V1 architecture is now the default in vLLM v0.31.0. It brings a new scheduler, chunked prefill enabled by default, and improved GPU memory management. If you see behavioral differences from older guides, V1 is likely the cause.

KV cache tuning

The KV cache stores key-value tensors for every token in every active sequence. It is usually the largest consumer of GPU memory after the model weights themselves, so tuning it has the biggest impact on how many concurrent requests you can serve.

Sizing formula

vLLM calculates the KV cache size automatically, but understanding the formula helps you set limits:

KV memory per token = 2 × num_layers × num_kv_heads × head_dim × dtype_bytes

The factor of 2 accounts for both keys and values. For a 7B model with 32 layers, 32 KV heads, 128 head dimension, and float16:

2 × 32 × 32 × 128 × 2 = 524,288 bytes ≈ 0.5 MB per token

With 24 GB of KV cache space, that holds roughly 48,000 tokens across all sequences.

Use fp8 KV cache

--kv-cache-dtype fp8 cuts KV cache memory by roughly 50% compared to the default float16. That directly doubles the number of tokens you can cache, which means more concurrent sequences or longer contexts.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8

FP8 KV cache introduces a small accuracy loss. For most chat and code-generation workloads the difference is negligible, but if you run evals on your specific task, verify that output quality meets your threshold before enabling it in production.

`--max-num-seqs` vs `--max-model-len`

These 2 flags compete for the same KV cache pool:

--max-model-len caps the context length of a single sequence. Lowering it frees KV cache for more concurrent sequences.
--max-num-seqs caps the number of sequences processed in one batch. Raising it increases throughput at the cost of per-sequence memory.

If your workload sends short prompts (under 2K tokens), lower --max-model-len and raise --max-num-seqs to maximize concurrency. If you serve long-context requests, do the opposite.

# Short prompts, high concurrency
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --max-num-seqs 256

# Long context, fewer concurrent sequences
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 32768 \
  --max-num-seqs 32

Prefix caching

Many real-world workloads send requests that share a common prefix: a system prompt, a RAG retrieval block, or earlier turns in a multi-turn conversation. Without prefix caching, vLLM recomputes the KV cache for that shared prefix on every request. --enable-prefix-caching stores computed KV blocks in a hash-indexed table. When a new request arrives with a prefix that matches a cached block, vLLM reuses the cached KV tensors instead of recomputing them.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching

Prefix caching is especially effective when:

Every request includes the same system prompt (common in chat apps and coding agents).
You use RAG with a fixed retrieval block prepended to each query.
Multi-turn conversations reuse earlier turns as context.

It provides no benefit when every request has a unique prefix.

Prefix caching is enabled by default in V1. You only need the flag explicitly if you run with V0 scheduler mode.

Chunked prefill

Prefill — processing the input prompt — is compute-heavy. A long prompt (8K+ tokens) can monopolize the GPU for hundreds of milliseconds, blocking decode steps for other sequences and causing latency spikes. Chunked prefill splits the prefill into smaller chunks interleaved with decode steps. The GPU processes a chunk of the prompt, then runs a decode step for other active sequences, then continues with the next prefill chunk. This keeps decode latency predictable. In V1, chunked prefill is enabled by default. You can control the chunk size:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-batched-tokens 8192

--max-num-batched-tokens sets the maximum number of tokens processed in a single scheduler step. Lower values reduce prefill impact on decode latency but increase total time-to-first-token (TTFT). Higher values improve prefill throughput at the cost of occasional decode jitter.

If you disabled chunked prefill in an older vLLM version and are upgrading to V1, remove the --enable-chunked-prefill flag — it is always on in V1 and the flag is ignored.

LoRA serving

vLLM can serve multiple LoRA adapters from a single process without merging them into the base model. This lets you run one base model and route requests to different fine-tuned variants — for example, a coding adapter, a summarization adapter, and a general-chat adapter — all sharing the same base weights in GPU memory.

Enable LoRA serving

vllm serve meta-llama/Llama-3.1-8B \
  --enable-lora \
  --lora-modules \
    coder=./lora-adapter-coder \
    summarizer=./lora-adapter-summarizer

Each entry in --lora-modules maps a request-facing name to a local path. Clients select the adapter in the model field:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "coder",
    "messages": [
      { "role": "user", "content": "Write a Python function that sorts a list." }
    ]
  }'

Dynamic loading

vLLM supports dynamic LoRA loading — you can add or remove adapters at runtime through the API without restarting the server. This is useful when you have many adapters but only a few are active at any time.

# Load a new adapter at runtime
curl http://localhost:8000/v1/load_lora_adapter \
  -H "Content-Type: application/json" \
  -d '{
    "lora_name": "translator",
    "lora_path": "./lora-adapter-translator"
  }'

Use --max-lora-rank to limit the rank of LoRA adapters the server accepts. Lower-rank adapters use less memory. If all your adapters share the same rank, set this to that rank to avoid over-allocating memory for unused capacity.

Quantization options

Quantization reduces model weight precision below 16-bit, trading a small amount of accuracy for significant memory and speed gains. vLLM supports several quantization formats:

Format	Precision	Hardware	Best for
AWQ	4-bit integer	Any GPU	General-purpose 4-bit quantization. Good accuracy retention. Pre-quantized models widely available on Hugging Face.
GPTQ	4-bit integer	Any GPU	Older format, similar to AWQ. Prefer AWQ for new quantizations — GPTQ remains for compatibility with existing models.
FP8	8-bit float	Ampere+ (A100, H100, L40, RTX 4090)	Near-FP16 accuracy with half the memory. Good default when your GPU supports it.
NVFP4	4-bit float	Blackwell (B200, GB200, RTX 5090)	Newest format. Maximum memory savings on Blackwell hardware.

Using quantized models

For pre-quantized models, vLLM detects the format automatically from the model’s config.json:

# AWQ model — auto-detected
vllm serve TheBloke/Llama-2-7B-Chat-AWQ

# FP8 model — auto-detected on Ampere+ GPUs
vllm serve neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8

FP8 shorthand changed in v0.31. The FP8 quantization identifier is now fp8_per_tensor. If you pass --quantization fp8 explicitly in a script or CI pipeline, update it to the new identifier. Models with FP8 in their config.json are unaffected — auto-detection still works.

Online quantization

You can also quantize a model on the fly during loading, though pre-quantized models are preferred for production:

vllm serve meta-llama/Llama-3.1-70B \
  --quantization awq

Throughput vs latency

Your optimization strategy depends on whether you prioritize throughput (total tokens per second across all requests) or latency (time-to-first-token and inter-token latency for individual requests).

Maximize throughput

Increase batch sizes to pack more work per GPU step:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 256 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8

--max-num-seqs 256 allows up to 256 concurrent sequences in a batch.
--max-num-batched-tokens 16384 allows more tokens per scheduler step.
--kv-cache-dtype fp8 frees memory for more sequences.

Minimize latency

Reduce batch sizes and disable optimizations that trade latency for throughput:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --enforce-eager

--max-num-seqs 16 keeps batches small so each request gets GPU time faster.
--enforce-eager disables CUDA graphs. CUDA graphs capture and replay a fixed computation graph, which improves throughput but adds latency on the first run of each new batch shape. Use --enforce-eager for debugging or when every millisecond of TTFT matters.

For interactive applications where users wait for a response, target TTFT under 500ms. Start with --max-num-seqs 32 and --max-num-batched-tokens 4096, then adjust based on your measured TTFT under load.

V1 architecture changes

V1 is the default scheduler in vLLM v0.31.0. If you used vLLM before, here is what changed: New scheduler. The V1 scheduler replaces the V0 block manager with a unified memory manager that handles KV cache, prefix caching, and chunked prefill in a single pass. This reduces scheduling overhead and eliminates several edge cases where V0 could fragment GPU memory. Chunked prefill is default. Prefill and decode are interleaved automatically. You no longer need --enable-chunked-prefill — it is always on. Better memory management. V1 tracks GPU memory at a finer granularity, reducing wasted space from padding and alignment. Expect 5-15% more usable KV cache on the same hardware compared to V0. CUDA graph improvements. V1 captures CUDA graphs more aggressively across a wider range of batch shapes, improving throughput without manual tuning.

If you encounter a regression with V1, you can fall back to V0 by setting the environment variable VLLM_USE_V1=0 before starting the server. Report the issue to the vLLM team — V0 will not receive new features.

New in v0.31: persistent weight cache and snapshots

vLLM v0.31 introduces 2 new CLI commands:

vllm preload starts a persistent weight-cache daemon that keeps model weights in GPU memory between server restarts. Instead of reloading a multi-gigabyte model from disk on every restart, the server connects to the daemon and reuses the cached weights. This cuts cold-start time from minutes to seconds.
vllm snapshot create captures the current server state (loaded model, adapters, configuration) and vllm snapshot restore restores it instantly. Useful for rolling back after a bad config change or replicating a known-good setup across machines.

# Start the weight-cache daemon
vllm preload meta-llama/Llama-3.1-8B-Instruct

# Create a snapshot of the current state
vllm snapshot create my-production-config

# Restore it on another machine
vllm snapshot restore my-production-config

Breaking changes in v0.31

--trust-request-mm-kwargs is now required when clients send multimodal inputs (images, audio) via the request body. Without this flag, vLLM rejects multimodal kwargs for security. Add it if your application sends images or other media in API requests.
--grpc enables gRPC serving alongside the HTTP server. If you use gRPC clients, start vLLM with this flag to expose the gRPC endpoint.

Example configurations

High-throughput batch processing

For offline batch workloads where total tokens per second matters more than individual request latency:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-seqs 512 \
  --max-num-batched-tokens 32768 \
  --max-model-len 8192

This configuration maximizes concurrency on a 4-GPU setup. The 4-bit-equivalent KV cache and high batch limits push throughput at the cost of longer per-request latency.

Low-latency interactive serving

For chat applications where users wait for responses and TTFT should be under 500ms:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-seqs 32 \
  --max-num-batched-tokens 4096 \
  --max-model-len 8192

Small batch sizes and moderate token limits keep the GPU responsive. Prefix caching accelerates repeated system prompts without adding latency.

Multi-adapter LoRA serving

For serving multiple fine-tuned variants from one base model:

vllm serve meta-llama/Llama-3.1-8B \
  --enable-lora \
  --lora-modules \
    coder=./adapters/coder \
    reviewer=./adapters/reviewer \
    summarizer=./adapters/summarizer \
  --max-lora-rank 64 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-seqs 128

Clients select the adapter by name in the model field. All 3 adapters share the same base model weights, so GPU memory usage is close to serving the base model alone — each LoRA adapter adds only a few megabytes.

After changing vLLM configuration, re-register your Tokios deployment so the connector picks up the new upstream endpoint. See Register Model.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Optimize vLLM for throughput and latency

KV cache tuning

Sizing formula

Use fp8 KV cache

`--max-num-seqs` vs `--max-model-len`

Prefix caching

Chunked prefill

LoRA serving

Enable LoRA serving

Dynamic loading

Quantization options

Using quantized models

Online quantization

Throughput vs latency

Maximize throughput

Minimize latency

V1 architecture changes

New in v0.31: persistent weight cache and snapshots

Breaking changes in v0.31

Example configurations

High-throughput batch processing

Low-latency interactive serving

Multi-adapter LoRA serving

​KV cache tuning

​Sizing formula

​Use fp8 KV cache

​--max-num-seqs vs --max-model-len

​Prefix caching

​Chunked prefill

​LoRA serving

​Enable LoRA serving

​Dynamic loading

​Quantization options

​Using quantized models

​Online quantization

​Throughput vs latency

​Maximize throughput

​Minimize latency

​V1 architecture changes

​New in v0.31: persistent weight cache and snapshots

​Breaking changes in v0.31

​Example configurations

​High-throughput batch processing

​Low-latency interactive serving

​Multi-adapter LoRA serving

KV cache tuning

Sizing formula

Use fp8 KV cache

`--max-num-seqs` vs `--max-model-len`

Prefix caching

Chunked prefill

LoRA serving

Enable LoRA serving

Dynamic loading

Quantization options

Using quantized models

Online quantization

Throughput vs latency

Maximize throughput

Minimize latency

V1 architecture changes

New in v0.31: persistent weight cache and snapshots

Breaking changes in v0.31

Example configurations

High-throughput batch processing

Low-latency interactive serving

Multi-adapter LoRA serving