> ## Documentation Index > Fetch the complete documentation index at: https://docs.tokios.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel. > Use 'connector' for the local tokios-connector process. > Use 'deployment' for the public model name in the API model field. > Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend. > API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-. # SGLang configuration and when to use it > Configure SGLang for RAG and agent workloads — RadixAttention for prefix caching, vLLM-vs-SGLang decision matrix, and SGLang-specific optimization flags. SGLang is a high-performance serving runtime built for throughput. Its signature feature, RadixAttention, uses a tree-structured KV cache that reuses computation across requests with shared token prefixes. That makes SGLang the strongest open-source choice for workloads where many requests carry the same system prompt, tool schema, or retrieved context — RAG pipelines, multi-turn chat, and agent loops. This guide covers when to pick SGLang over vLLM, the configuration flags that matter, and ready-to-use launch commands for common setups. > [!NOTE] > > Last verified: 2026-10-14. This page reflects SGLang **v0.5.21** (released October 2, 2026). Flag names and defaults can shift between releases — verify against your installed version with `python -m sglang.launch_server --help`. ## RadixAttention SGLang's KV cache is not a flat table — it is a **radix tree** keyed by token sequences. Every prefix any request has ever processed lives in the tree. When a new request arrives, SGLang walks the tree to find the longest matching prefix and reuses its cached KV state instead of recomputing it. Concretely, if 10 requests share a 2,000-token system prompt, the prompt is prefilled once. The remaining 9 requests skip straight to decode. As conversations diverge, each branch gets its own cache node, and shared prefixes at any depth are reused independently. The performance difference is significant for prefix-heavy workloads. At 50 concurrent requests with 80% shared prefix, SGLang delivers **37% lower time-to-first-token (TTFT)** compared to vLLM. The advantage grows as prefix overlap increases and shrinks when every request is unique. > [!TIP] > > RadixAttention is not something you enable — it is how SGLang's KV cache works by default. Every request benefits automatically. The configuration flags control eviction policy, memory limits, and the Rust-backed storage backend. ## When SGLang wins Pick SGLang when your workload has **high prefix overlap**: * **RAG pipelines** — the same system prompt and retrieved chunks appear across many queries. RadixAttention caches the prompt + document prefix once and reuses it for every question. * **Multi-turn conversations** — each turn extends the previous context. The entire conversation history up to the last turn is a shared prefix. * **Agent loops with tool schemas** — agents send the same tool definitions, few-shot examples, and orchestration prompts on every call. SGLang caches those prefixes and only prefills the new tool results. * **Long shared system prompts** — any deployment where dozens or hundreds of users share a large system prompt (customer support bots, coding assistants, document Q\&A). * **Batch evaluation** — running the same prompt template with different variable substitutions. The template prefix is cached; only the filled-in portion is prefilled. ## When vLLM wins Pick vLLM when your workload prioritizes **breadth over depth**: * **Unique prompts per request** — if every request has a completely different prompt with no shared prefix, RadixAttention has nothing to reuse. vLLM's PagedAttention handles independent requests equally well. * **Broad model support** — vLLM adds support for new model architectures faster. If you need a model that shipped last week, vLLM likely supports it first. * **Blackwell GPU optimization** — vLLM has dedicated optimization paths for NVIDIA Blackwell (B100/B200/GB200) hardware. If you are deploying on Blackwell, vLLM extracts more throughput. * **EAGLE3 speculative decoding** — vLLM's EAGLE integration is more mature. If speculative decoding is your primary latency lever, vLLM is the safer choice today. * **Production hardening** — vLLM has a larger deployment base and longer track record in production serving. If stability and community support matter more than prefix caching, vLLM is the default. ## Configuration flags Key SGLang-specific flags for `python -m sglang.launch_server`: | Flag | Default | Purpose | | - | - | - | | `--model-path` | (required) | HuggingFace repo id or local path to the model | | `--port` | `30000` | Listen port for the OpenAI-compatible server | | `--host` | `127.0.0.1` | Bind address — keep loopback unless a reverse proxy fronts SGLang | | `--tp` | `1` | Tensor parallelism across GPUs | | `--mem-fraction-static` | `0.88` | Fraction of GPU memory allocated to the KV cache at startup. Increase to `0.90`-`0.92` for long-context workloads if you have headroom. | | `--context-length` | model max | Override the model's maximum context length. Set this lower than the model max to reduce KV cache memory per request. | | `--chunked-prefill-size` | `-1` (auto) | Split large prefills into chunks to avoid stalling decode. Set to `2048`-`4096` for mixed workloads with long inputs. | | `--disable-radix-cache` | `false` | Disables RadixAttention entirely. You almost never want this — it removes SGLang's core advantage. | | `--enable-metrics` | `false` | Exposes Prometheus metrics at `/metrics`. Enable in production. | | `--api-key` | (none) | Require a bearer token on requests to SGLang's own server. Useful as defense in depth on loopback. | ### Rust-backed prefix caching Starting in **v0.5.21**, the RadixAttention cache backend is implemented in Rust and enabled by default. The Rust backend replaces the earlier Python implementation with lower overhead for cache lookups, insertions, and eviction. You do not need a flag to enable it — it is the default path. If you encounter issues, you can fall back to the Python implementation with `--disable-cuda-graph` and check the SGLang release notes for the exact rollback flag for your version. In practice, the Rust backend is stable and faster. ## Decision matrix | Workload | Recommended runtime | Why | | - | - | - | | RAG with repeated context | **SGLang** | RadixAttention caches system prompt + retrieved chunks | | Multi-turn chat (10+ turns) | **SGLang** | Each turn extends a shared prefix | | Agent loops with tool schemas | **SGLang** | Tool definitions cached after first call | | Long shared system prompt | **SGLang** | One prefill serves many requests | | Unique prompts per request | **vLLM** | No prefix reuse benefit; PagedAttention is equally fast | | Latest model architectures | **vLLM** | Faster adoption of new models | | Blackwell GPU (B100/B200) | **vLLM** | Dedicated Blackwell optimization paths | | EAGLE3 speculative decoding | **vLLM** | More mature speculative decoding support | | Batch evaluation (same template) | **SGLang** | Template prefix cached across all variants | | Single-user, low-concurrency | Either | Both perform well; pick by familiarity | ## Example configurations ### RAG pipeline with long retrieved context Serve a 70B model optimized for RAG, where every request carries the same system prompt and large retrieved document chunks: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-70B-Instruct \ --port 30000 \ --tp 4 \ --mem-fraction-static 0.92 \ --context-length 16384 \ --chunked-prefill-size 4096 \ --enable-metrics ``` `--mem-fraction-static 0.92` allocates more GPU memory to the KV cache for long retrieved contexts. `--chunked-prefill-size 4096` prevents large document prefills from stalling concurrent decode requests. RadixAttention handles the rest — the system prompt and document prefix are cached after the first request. ### Multi-turn chat agent with tool schemas Serve a model for an agent that sends the same tool definitions on every call: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --port 30000 \ --tp 1 \ --mem-fraction-static 0.90 \ --context-length 32768 \ --chunked-prefill-size 2048 \ --enable-metrics ``` The agent's tool schema, system prompt, and few-shot examples are identical across calls. RadixAttention caches that entire prefix. Only the new tool results and user message at the end of each turn require a fresh prefill. With 32K context, multi-turn conversations can run dozens of turns before hitting the limit. ### High-throughput batch evaluation Run the same prompt template across thousands of inputs with maximum throughput: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-70B-Instruct \ --port 30000 \ --tp 8 \ --mem-fraction-static 0.90 \ --context-length 8192 \ --chunked-prefill-size 4096 \ --enable-metrics ``` The prompt template is prefilled once and cached. Each request only prefills its unique variable substitution. Across 8 GPUs with tensor parallelism, the batch throughput is dominated by decode — prefill overhead is nearly eliminated. > [!WARNING] > > **Security advisory — Qwen VL video preprocessing.** SGLang through v0.5.21 has a memory exhaustion denial-of-service vulnerability in Qwen VL video preprocessing. A crafted request can trigger unbounded memory allocation during video frame decoding. If you serve Qwen VL models, apply the upstream patch or upgrade to a version that includes the fix. If you cannot upgrade immediately, restrict access to the SGLang server to trusted callers only. ## Native decisions API SGLang v0.5.21 added a **native decisions API** that follows the same pattern as Ollama and LM Studio. This lets clients query which models are available, check model status, and manage deployments through a consistent interface — useful if you build tooling that talks to multiple local backends. Check the SGLang release notes for the exact endpoint paths and request/response format, as the API surface is new and may evolve. ## Connect SGLang through Tokios Once SGLang is running, the Tokios connector exposes it to remote clients without opening any inbound ports. The connector dials out to `https://api.tokios.com` and forwards requests to SGLang's OpenAI-compatible `/v1` endpoint on localhost. See [SGLang remote access](/guides/sglang-remote-access.md) for the full setup walkthrough. > **[SGLang remote access](/guides/sglang-remote-access.md)** > > Connect SGLang through Tokios for authenticated remote access. > **[vLLM remote access](/guides/vllm-remote-access.md)** > > Same pattern for vLLM — compare setups side by side. > **[Speculative decoding](/guides/speculative-decoding.md)** > > Speed up decode with draft models, n-grams, MTP, and EAGLE. > **[Benchmark your deployment](/guides/benchmark-your-deployment.md)** > > Measure TTFT, throughput, and latency across configurations. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.