> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Optimize vLLM for throughput and latency

> Tune vLLM for maximum throughput or minimum latency — KV cache tuning, prefix caching, chunked prefill, LoRA serving, quantization options, and V1 architecture changes.

vLLM is the most feature-rich open-source runtime for serving large language models on GPUs. It ships with a wide range of configuration options that directly affect throughput, latency, and memory usage. This guide covers the knobs that matter most and how to set them for your workload.

> [!NOTE]
>
> Last verified: 2026-10-14 — vLLM v0.31.0.

> [!NOTE]
>
> **V1 is the default.** The V1 architecture is now the default in vLLM v0.31.0. It brings a new scheduler, chunked prefill enabled by default, and improved GPU memory management. If you see behavioral differences from older guides, V1 is likely the cause.

## KV cache tuning

The KV cache stores key-value tensors for every token in every active sequence. It is usually the largest consumer of GPU memory after the model weights themselves, so tuning it has the biggest impact on how many concurrent requests you can serve.

### Sizing formula

vLLM calculates the KV cache size automatically, but understanding the formula helps you set limits:

```
KV memory per token = 2 × num_layers × num_kv_heads × head_dim × dtype_bytes
```

The factor of 2 accounts for both keys and values. For a 7B model with 32 layers, 32 KV heads, 128 head dimension, and `float16`:

```
2 × 32 × 32 × 128 × 2 = 524,288 bytes ≈ 0.5 MB per token
```

With 24 GB of KV cache space, that holds roughly 48,000 tokens across all sequences.

### Use fp8 KV cache

`--kv-cache-dtype fp8` cuts KV cache memory by roughly 50% compared to the default `float16`. That directly doubles the number of tokens you can cache, which means more concurrent sequences or longer contexts.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8
```

> [!WARNING]
>
> FP8 KV cache introduces a small accuracy loss. For most chat and code-generation workloads the difference is negligible, but if you run evals on your specific task, verify that output quality meets your threshold before enabling it in production.

### `--max-num-seqs` vs `--max-model-len`

These 2 flags compete for the same KV cache pool:

* `--max-model-len` caps the context length of a single sequence. Lowering it frees KV cache for more concurrent sequences.
* `--max-num-seqs` caps the number of sequences processed in one batch. Raising it increases throughput at the cost of per-sequence memory.

If your workload sends short prompts (under 2K tokens), lower `--max-model-len` and raise `--max-num-seqs` to maximize concurrency. If you serve long-context requests, do the opposite.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Short prompts, high concurrency
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --max-num-seqs 256

# Long context, fewer concurrent sequences
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 32768 \
  --max-num-seqs 32
```

## Prefix caching

Many real-world workloads send requests that share a common prefix: a system prompt, a RAG retrieval block, or earlier turns in a multi-turn conversation. Without prefix caching, vLLM recomputes the KV cache for that shared prefix on every request.

`--enable-prefix-caching` stores computed KV blocks in a hash-indexed table. When a new request arrives with a prefix that matches a cached block, vLLM reuses the cached KV tensors instead of recomputing them.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching
```

> [!TIP]
>
> Prefix caching is especially effective when:
>
> * Every request includes the same system prompt (common in chat apps and coding agents).
> * You use RAG with a fixed retrieval block prepended to each query.
> * Multi-turn conversations reuse earlier turns as context.
>
> It provides no benefit when every request has a unique prefix.

Prefix caching is enabled by default in V1. You only need the flag explicitly if you run with `V0` scheduler mode.

## Chunked prefill

Prefill — processing the input prompt — is compute-heavy. A long prompt (8K+ tokens) can monopolize the GPU for hundreds of milliseconds, blocking decode steps for other sequences and causing latency spikes.

Chunked prefill splits the prefill into smaller chunks interleaved with decode steps. The GPU processes a chunk of the prompt, then runs a decode step for other active sequences, then continues with the next prefill chunk. This keeps decode latency predictable.

In V1, chunked prefill is **enabled by default**. You can control the chunk size:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-batched-tokens 8192
```

`--max-num-batched-tokens` sets the maximum number of tokens processed in a single scheduler step. Lower values reduce prefill impact on decode latency but increase total time-to-first-token (TTFT). Higher values improve prefill throughput at the cost of occasional decode jitter.

> [!NOTE]
>
> If you disabled chunked prefill in an older vLLM version and are upgrading to V1, remove the `--enable-chunked-prefill` flag — it is always on in V1 and the flag is ignored.

## LoRA serving

vLLM can serve multiple LoRA adapters from a single process without merging them into the base model. This lets you run one base model and route requests to different fine-tuned variants — for example, a coding adapter, a summarization adapter, and a general-chat adapter — all sharing the same base weights in GPU memory.

### Enable LoRA serving

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B \
  --enable-lora \
  --lora-modules \
    coder=./lora-adapter-coder \
    summarizer=./lora-adapter-summarizer
```

Each entry in `--lora-modules` maps a request-facing name to a local path. Clients select the adapter in the `model` field:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "coder",
    "messages": [
      { "role": "user", "content": "Write a Python function that sorts a list." }
    ]
  }'
```

### Dynamic loading

vLLM supports dynamic LoRA loading — you can add or remove adapters at runtime through the API without restarting the server. This is useful when you have many adapters but only a few are active at any time.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Load a new adapter at runtime
curl http://localhost:8000/v1/load_lora_adapter \
  -H "Content-Type: application/json" \
  -d '{
    "lora_name": "translator",
    "lora_path": "./lora-adapter-translator"
  }'
```

> [!TIP]
>
> Use `--max-lora-rank` to limit the rank of LoRA adapters the server accepts. Lower-rank adapters use less memory. If all your adapters share the same rank, set this to that rank to avoid over-allocating memory for unused capacity.

## Quantization options

Quantization reduces model weight precision below 16-bit, trading a small amount of accuracy for significant memory and speed gains. vLLM supports several quantization formats:

| Format | Precision | Hardware | Best for |
| - | - | - | - |
| **AWQ** | 4-bit integer | Any GPU | General-purpose 4-bit quantization. Good accuracy retention. Pre-quantized models widely available on Hugging Face. |
| **GPTQ** | 4-bit integer | Any GPU | Older format, similar to AWQ. Prefer AWQ for new quantizations — GPTQ remains for compatibility with existing models. |
| **FP8** | 8-bit float | Ampere+ (A100, H100, L40, RTX 4090) | Near-FP16 accuracy with half the memory. Good default when your GPU supports it. |
| **NVFP4** | 4-bit float | Blackwell (B200, GB200, RTX 5090) | Newest format. Maximum memory savings on Blackwell hardware. |

### Using quantized models

For pre-quantized models, vLLM detects the format automatically from the model's `config.json`:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# AWQ model — auto-detected
vllm serve TheBloke/Llama-2-7B-Chat-AWQ

# FP8 model — auto-detected on Ampere+ GPUs
vllm serve neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8
```

> [!WARNING]
>
> **FP8 shorthand changed in v0.31.** The FP8 quantization identifier is now `fp8_per_tensor`. If you pass `--quantization fp8` explicitly in a script or CI pipeline, update it to the new identifier. Models with FP8 in their `config.json` are unaffected — auto-detection still works.

### Online quantization

You can also quantize a model on the fly during loading, though pre-quantized models are preferred for production:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B \
  --quantization awq
```

## Throughput vs latency

Your optimization strategy depends on whether you prioritize **throughput** (total tokens per second across all requests) or **latency** (time-to-first-token and inter-token latency for individual requests).

### Maximize throughput

Increase batch sizes to pack more work per GPU step:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 256 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8
```

* `--max-num-seqs 256` allows up to 256 concurrent sequences in a batch.
* `--max-num-batched-tokens 16384` allows more tokens per scheduler step.
* `--kv-cache-dtype fp8` frees memory for more sequences.

### Minimize latency

Reduce batch sizes and disable optimizations that trade latency for throughput:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --enforce-eager
```

* `--max-num-seqs 16` keeps batches small so each request gets GPU time faster.
* `--enforce-eager` disables CUDA graphs. CUDA graphs capture and replay a fixed computation graph, which improves throughput but adds latency on the first run of each new batch shape. Use `--enforce-eager` for debugging or when every millisecond of TTFT matters.

> [!TIP]
>
> For interactive applications where users wait for a response, target TTFT under 500ms. Start with `--max-num-seqs 32` and `--max-num-batched-tokens 4096`, then adjust based on your measured TTFT under load.

## V1 architecture changes

V1 is the default scheduler in vLLM v0.31.0. If you used vLLM before, here is what changed:

**New scheduler.** The V1 scheduler replaces the V0 block manager with a unified memory manager that handles KV cache, prefix caching, and chunked prefill in a single pass. This reduces scheduling overhead and eliminates several edge cases where V0 could fragment GPU memory.

**Chunked prefill is default.** Prefill and decode are interleaved automatically. You no longer need `--enable-chunked-prefill` — it is always on.

**Better memory management.** V1 tracks GPU memory at a finer granularity, reducing wasted space from padding and alignment. Expect 5-15% more usable KV cache on the same hardware compared to V0.

**CUDA graph improvements.** V1 captures CUDA graphs more aggressively across a wider range of batch shapes, improving throughput without manual tuning.

> [!NOTE]
>
> If you encounter a regression with V1, you can fall back to V0 by setting the environment variable `VLLM_USE_V1=0` before starting the server. Report the issue to the vLLM team — V0 will not receive new features.

### New in v0.31: persistent weight cache and snapshots

vLLM v0.31 introduces 2 new CLI commands:

* **`vllm preload`** starts a persistent weight-cache daemon that keeps model weights in GPU memory between server restarts. Instead of reloading a multi-gigabyte model from disk on every restart, the server connects to the daemon and reuses the cached weights. This cuts cold-start time from minutes to seconds.

* **`vllm snapshot create`** captures the current server state (loaded model, adapters, configuration) and **`vllm snapshot restore`** restores it instantly. Useful for rolling back after a bad config change or replicating a known-good setup across machines.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Start the weight-cache daemon
vllm preload meta-llama/Llama-3.1-8B-Instruct

# Create a snapshot of the current state
vllm snapshot create my-production-config

# Restore it on another machine
vllm snapshot restore my-production-config
```

### Breaking changes in v0.31

* **`--trust-request-mm-kwargs`** is now required when clients send multimodal inputs (images, audio) via the request body. Without this flag, vLLM rejects multimodal kwargs for security. Add it if your application sends images or other media in API requests.

* **`--grpc`** enables gRPC serving alongside the HTTP server. If you use gRPC clients, start vLLM with this flag to expose the gRPC endpoint.

## Example configurations

### High-throughput batch processing

For offline batch workloads where total tokens per second matters more than individual request latency:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-seqs 512 \
  --max-num-batched-tokens 32768 \
  --max-model-len 8192
```

This configuration maximizes concurrency on a 4-GPU setup. The 4-bit-equivalent KV cache and high batch limits push throughput at the cost of longer per-request latency.

### Low-latency interactive serving

For chat applications where users wait for responses and TTFT should be under 500ms:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-seqs 32 \
  --max-num-batched-tokens 4096 \
  --max-model-len 8192
```

Small batch sizes and moderate token limits keep the GPU responsive. Prefix caching accelerates repeated system prompts without adding latency.

### Multi-adapter LoRA serving

For serving multiple fine-tuned variants from one base model:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
vllm serve meta-llama/Llama-3.1-8B \
  --enable-lora \
  --lora-modules \
    coder=./adapters/coder \
    reviewer=./adapters/reviewer \
    summarizer=./adapters/summarizer \
  --max-lora-rank 64 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --max-num-seqs 128
```

Clients select the adapter by name in the `model` field. All 3 adapters share the same base model weights, so GPU memory usage is close to serving the base model alone — each LoRA adapter adds only a few megabytes.

> [!NOTE]
>
> After changing vLLM configuration, re-register your Tokios deployment so the connector picks up the new upstream endpoint. See [Register Model](/setup/register-model.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.