SGLang configuration and when to use it

SGLang is a high-performance serving runtime built for throughput. Its signature feature, RadixAttention, uses a tree-structured KV cache that reuses computation across requests with shared token prefixes. That makes SGLang the strongest open-source choice for workloads where many requests carry the same system prompt, tool schema, or retrieved context — RAG pipelines, multi-turn chat, and agent loops. This guide covers when to pick SGLang over vLLM, the configuration flags that matter, and ready-to-use launch commands for common setups.

Last verified: 2026-10-14. This page reflects SGLang v0.5.21 (released October 2, 2026). Flag names and defaults can shift between releases — verify against your installed version with python -m sglang.launch_server --help.

RadixAttention

SGLang’s KV cache is not a flat table — it is a radix tree keyed by token sequences. Every prefix any request has ever processed lives in the tree. When a new request arrives, SGLang walks the tree to find the longest matching prefix and reuses its cached KV state instead of recomputing it. Concretely, if 10 requests share a 2,000-token system prompt, the prompt is prefilled once. The remaining 9 requests skip straight to decode. As conversations diverge, each branch gets its own cache node, and shared prefixes at any depth are reused independently. The performance difference is significant for prefix-heavy workloads. At 50 concurrent requests with 80% shared prefix, SGLang delivers 37% lower time-to-first-token (TTFT) compared to vLLM. The advantage grows as prefix overlap increases and shrinks when every request is unique.

RadixAttention is not something you enable — it is how SGLang’s KV cache works by default. Every request benefits automatically. The configuration flags control eviction policy, memory limits, and the Rust-backed storage backend.

When SGLang wins

Pick SGLang when your workload has high prefix overlap:

RAG pipelines — the same system prompt and retrieved chunks appear across many queries. RadixAttention caches the prompt + document prefix once and reuses it for every question.
Multi-turn conversations — each turn extends the previous context. The entire conversation history up to the last turn is a shared prefix.
Agent loops with tool schemas — agents send the same tool definitions, few-shot examples, and orchestration prompts on every call. SGLang caches those prefixes and only prefills the new tool results.
Long shared system prompts — any deployment where dozens or hundreds of users share a large system prompt (customer support bots, coding assistants, document Q&A).
Batch evaluation — running the same prompt template with different variable substitutions. The template prefix is cached; only the filled-in portion is prefilled.

When vLLM wins

Pick vLLM when your workload prioritizes breadth over depth:

Unique prompts per request — if every request has a completely different prompt with no shared prefix, RadixAttention has nothing to reuse. vLLM’s PagedAttention handles independent requests equally well.
Broad model support — vLLM adds support for new model architectures faster. If you need a model that shipped last week, vLLM likely supports it first.
Blackwell GPU optimization — vLLM has dedicated optimization paths for NVIDIA Blackwell (B100/B200/GB200) hardware. If you are deploying on Blackwell, vLLM extracts more throughput.
EAGLE3 speculative decoding — vLLM’s EAGLE integration is more mature. If speculative decoding is your primary latency lever, vLLM is the safer choice today.
Production hardening — vLLM has a larger deployment base and longer track record in production serving. If stability and community support matter more than prefix caching, vLLM is the default.

Configuration flags

Key SGLang-specific flags for python -m sglang.launch_server:

Flag	Default	Purpose
`--model-path`	(required)	HuggingFace repo id or local path to the model
`--port`	`30000`	Listen port for the OpenAI-compatible server
`--host`	`127.0.0.1`	Bind address — keep loopback unless a reverse proxy fronts SGLang
`--tp`	`1`	Tensor parallelism across GPUs
`--mem-fraction-static`	`0.88`	Fraction of GPU memory allocated to the KV cache at startup. Increase to `0.90`-`0.92` for long-context workloads if you have headroom.
`--context-length`	model max	Override the model’s maximum context length. Set this lower than the model max to reduce KV cache memory per request.
`--chunked-prefill-size`	`-1` (auto)	Split large prefills into chunks to avoid stalling decode. Set to `2048`-`4096` for mixed workloads with long inputs.
`--disable-radix-cache`	`false`	Disables RadixAttention entirely. You almost never want this — it removes SGLang’s core advantage.
`--enable-metrics`	`false`	Exposes Prometheus metrics at `/metrics`. Enable in production.
`--api-key`	(none)	Require a bearer token on requests to SGLang’s own server. Useful as defense in depth on loopback.

Rust-backed prefix caching

Starting in v0.5.21, the RadixAttention cache backend is implemented in Rust and enabled by default. The Rust backend replaces the earlier Python implementation with lower overhead for cache lookups, insertions, and eviction. You do not need a flag to enable it — it is the default path. If you encounter issues, you can fall back to the Python implementation with --disable-cuda-graph and check the SGLang release notes for the exact rollback flag for your version. In practice, the Rust backend is stable and faster.

Decision matrix

Workload	Recommended runtime	Why
RAG with repeated context	SGLang	RadixAttention caches system prompt + retrieved chunks
Multi-turn chat (10+ turns)	SGLang	Each turn extends a shared prefix
Agent loops with tool schemas	SGLang	Tool definitions cached after first call
Long shared system prompt	SGLang	One prefill serves many requests
Unique prompts per request	vLLM	No prefix reuse benefit; PagedAttention is equally fast
Latest model architectures	vLLM	Faster adoption of new models
Blackwell GPU (B100/B200)	vLLM	Dedicated Blackwell optimization paths
EAGLE3 speculative decoding	vLLM	More mature speculative decoding support
Batch evaluation (same template)	SGLang	Template prefix cached across all variants
Single-user, low-concurrency	Either	Both perform well; pick by familiarity

Example configurations

RAG pipeline with long retrieved context

Serve a 70B model optimized for RAG, where every request carries the same system prompt and large retrieved document chunks:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --port 30000 \
  --tp 4 \
  --mem-fraction-static 0.92 \
  --context-length 16384 \
  --chunked-prefill-size 4096 \
  --enable-metrics

--mem-fraction-static 0.92 allocates more GPU memory to the KV cache for long retrieved contexts. --chunked-prefill-size 4096 prevents large document prefills from stalling concurrent decode requests. RadixAttention handles the rest — the system prompt and document prefix are cached after the first request.

Multi-turn chat agent with tool schemas

Serve a model for an agent that sends the same tool definitions on every call:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000 \
  --tp 1 \
  --mem-fraction-static 0.90 \
  --context-length 32768 \
  --chunked-prefill-size 2048 \
  --enable-metrics

The agent’s tool schema, system prompt, and few-shot examples are identical across calls. RadixAttention caches that entire prefix. Only the new tool results and user message at the end of each turn require a fresh prefill. With 32K context, multi-turn conversations can run dozens of turns before hitting the limit.

High-throughput batch evaluation

Run the same prompt template across thousands of inputs with maximum throughput:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --port 30000 \
  --tp 8 \
  --mem-fraction-static 0.90 \
  --context-length 8192 \
  --chunked-prefill-size 4096 \
  --enable-metrics

The prompt template is prefilled once and cached. Each request only prefills its unique variable substitution. Across 8 GPUs with tensor parallelism, the batch throughput is dominated by decode — prefill overhead is nearly eliminated.

Security advisory — Qwen VL video preprocessing. SGLang through v0.5.21 has a memory exhaustion denial-of-service vulnerability in Qwen VL video preprocessing. A crafted request can trigger unbounded memory allocation during video frame decoding. If you serve Qwen VL models, apply the upstream patch or upgrade to a version that includes the fix. If you cannot upgrade immediately, restrict access to the SGLang server to trusted callers only.

Native decisions API

SGLang v0.5.21 added a native decisions API that follows the same pattern as Ollama and LM Studio. This lets clients query which models are available, check model status, and manage deployments through a consistent interface — useful if you build tooling that talks to multiple local backends. Check the SGLang release notes for the exact endpoint paths and request/response format, as the API surface is new and may evolve.

Connect SGLang through Tokios

Once SGLang is running, the Tokios connector exposes it to remote clients without opening any inbound ports. The connector dials out to https://api.tokios.com and forwards requests to SGLang’s OpenAI-compatible /v1 endpoint on localhost. See SGLang remote access for the full setup walkthrough.

SGLang remote access

Connect SGLang through Tokios for authenticated remote access.

vLLM remote access

Same pattern for vLLM — compare setups side by side.

Speculative decoding

Speed up decode with draft models, n-grams, MTP, and EAGLE.

Benchmark your deployment

Measure TTFT, throughput, and latency across configurations.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

SGLang configuration and when to use it

RadixAttention

When SGLang wins

When vLLM wins

Configuration flags

Rust-backed prefix caching

Decision matrix

Example configurations

RAG pipeline with long retrieved context

Multi-turn chat agent with tool schemas

High-throughput batch evaluation

Native decisions API

Connect SGLang through Tokios

SGLang remote access

vLLM remote access

Speculative decoding

Benchmark your deployment

​RadixAttention

​When SGLang wins

​When vLLM wins

​Configuration flags

​Rust-backed prefix caching

​Decision matrix

​Example configurations

​RAG pipeline with long retrieved context

​Multi-turn chat agent with tool schemas

​High-throughput batch evaluation

​Native decisions API

​Connect SGLang through Tokios

SGLang remote access

vLLM remote access

Speculative decoding

Benchmark your deployment

RadixAttention

When SGLang wins

When vLLM wins

Configuration flags

Rust-backed prefix caching

Decision matrix

Example configurations

RAG pipeline with long retrieved context

Multi-turn chat agent with tool schemas

High-throughput batch evaluation

Native decisions API

Connect SGLang through Tokios