Run vLLM locally and access it from anywhere through Tokios

This page walks you through the full journey: from a fresh vLLM install on your GPU machine to a remotely accessible, authenticated API endpoint — in about 5 minutes. Your model stays on your hardware. Your prompts never pass through a third-party inference server.

Already have vLLM running? Skip to Step 3: Install the Tokios connector. If you just want the short version, see vLLM remote access.

Prerequisites

Before you start, make sure you have:

1 NVIDIA GPU with CUDA support (RTX 3090, 4090, A100, or DGX Spark).
CUDA 12.1+ installed (run nvidia-smi to check).
Python 3.9–3.12 with pip available.
A Tokios account (free to sign up).

For multi-GPU setups, see tensor parallelism below.

Step 1: Install and launch vLLM

Install vLLM with pip:

pip install vllm

Launch the OpenAI-compatible server. Keep it bound to 127.0.0.1 — don’t bind wider:

vllm serve mistralai/Mistral-7B-Instruct-v0.2 \
  --host 127.0.0.1 \
  --port 8000

vLLM downloads the model weights on first run, loads them into GPU memory, and starts an OpenAI-compatible HTTP server on http://127.0.0.1:8000.

Verify it’s running

In another terminal:

curl http://127.0.0.1:8000/v1/models

You should see your model listed.

Common launch configurations

Pick the flags that match your hardware:

vllm serve mistralai/Mistral-7B-Instruct-v0.2 \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --host 127.0.0.1 \
  --port 8000 \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096

vllm serve nvidia/Qwen3.5-35B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192

vllm serve meta-llama/Llama-3-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --kv-cache-dtype fp8 \
  --max-model-len 16384

--max-model-len controls how much GPU memory vLLM reserves for the KV cache. Start conservative (4096 or 8192) and increase once you confirm the model loads without OOM. See KV cache and context for the memory math.

Multi-GPU: tensor parallelism

For models that don’t fit in 1 GPU, use tensor parallelism across multiple GPUs on the same machine:

# 2 GPUs, tensor parallel
vllm serve meta-llama/Llama-3-70B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90

Tensor parallelism is preferred over pipeline parallelism for single-node multi-GPU — it gives better bandwidth utilization during the decode phase.

Windows (WSL2)

vLLM doesn’t run natively on Windows. Use Docker inside WSL2 with GPU passthrough:

# docker-compose.yml
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    command: >
      --model mistralai/Mistral-7B-Instruct-v0.2
      --host 0.0.0.0
      --port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

docker compose up

Step 2: Test locally

Before setting up remote access, confirm vLLM responds correctly:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "messages": [{"role": "user", "content": "Write hello world in Python"}],
    "max_tokens": 100
  }'

You should get a valid chat completion response.

Step 3: Install the Tokios connector

The connector dials out from your machine to https://api.tokios.com. No inbound port opens. No port forwarding. No DNS record. Sign in at tokios.com/console, open the Setup tab, and click Start pairing. Tokios shows a one-time claim code like TKS-XXXX-XXXX, valid for about 15 minutes. Run the install command on the machine running vLLM:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Already have tokios-connector installed? Pass the code directly:

tokios-connector TKS-XXXX-XXXX

Back in the console, click Approve on the Setup or Connectors tab. The connector’s token (ct-tok-…) is delivered over the tunnel once approved — you never see it on screen or paste it into a file.

Step 4: Point the connector at vLLM

vLLM’s OpenAI-compatible server speaks the exact surface Tokios expects. For a single model, the console’s Setup wizard defaults to http://127.0.0.1:8000/v1 automatically. For multiple models or a custom config, create a tokios-connector.json:

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8000/v1"
  }
}

See Connector config for the full multi-model file format.

Step 5: Register a deployment

In the Models tab, register your vLLM model and choose a public name — for example, mistral-tunnel. That public name is what clients send in the model field. It’s decoupled from vLLM’s local model id, so you can change the upstream model without breaking client configs.

Running LoRA adapters? Start vLLM with --enable-lora --lora-modules adapter1=/path/to/adapter, then register each adapter as a separate Tokios deployment. See Model routing.

Step 6: Create an API key

In the Keys tab, click Create key.

Optionally scope it to specific deployments with model-name patterns (comma-separated globs, * for all).
Copy the sk-tok-… key now — it’s shown only once.

You can create multiple keys — one per teammate, agent, or environment — and revoke any of them independently.

Step 7: Connect from any client

Your vLLM model is now reachable at https://api.tokios.com. Here’s how to call it:

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Write hello world in Rust"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="mistral-tunnel",
    messages=[{"role": "user", "content": "Write hello world in Rust"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model mistral-tunnel

export OPENAI_API_KEY=sk-tok-YOUR_KEY
aider --openai-api-base https://api.tokios.com/v1 \
      --model openai/mistral-tunnel

{
  "baseURL": "https://api.tokios.com/v1",
  "apiKey": "sk-tok-YOUR_KEY",
  "model": "mistral-tunnel"
}

The model field is always your registered deployment name (mistral-tunnel), never vLLM’s local model id.

Security model: outbound-only tunnel

Here’s what makes this different from SSH tunnels, port forwarding, or Tailscale:

Your GPU machine                  api.tokios.com               Remote client
──────────────                    ──────────────               ─────────────
vLLM ─┐                                                        sk-tok-… key
  (localhost:8000)                                               │
       ├─ connector ──► outbound TLS tunnel ──► gateway ──────► HTTPS
       │  (no inbound     (initiates             (auth +       request
       │   port open)      connection)            routing)
       └─ nothing listens
          on any port

Property	Your setup
Inbound ports open	0
Public IP required	No
Works behind CGNAT	Yes
Auth method	`sk-tok-…` bearer token per request
Key scoping	Per deployment, per user
Encryption	TLS end-to-end
SSE streaming	Native, no buffering

Your vLLM process never changes. It stays on 127.0.0.1:8000, unreachable from the outside. The connector is the only thing that can reach it, and the connector only dials out.

You can keep vLLM’s --api-key flag as defense in depth on the loopback connection between the connector and vLLM. Client requests through Tokios are authenticated with your sk-tok-… key — the connector is what’s allowed to reach 127.0.0.1:8000 at all.

FAQ

Does streaming work?

Yes. Set "stream": true and Tokios returns standard server-sent events (SSE) in OpenAI or Anthropic format, depending on which API surface you called.

Can I serve more than one model?

Yes. Register additional deployments in the Models tab, or use a tokios-connector.json config to define multiple upstreams and routes. See Connector config.

What happens if vLLM restarts?

The connector automatically reconnects to vLLM when it comes back up. No manual intervention needed — just restart vLLM and the endpoint recovers.

Does this work on Windows?

Yes. Run vLLM inside WSL2 with Docker (see the WSL2 config above), then install the Tokios connector on the Windows host or inside WSL2 — either works.

Your vLLM, accessible from anywhere

vLLM keeps running exactly as it does today — on localhost, on your GPU, with no inbound ports. The connector only dials out. Everything past that — authentication, routing, streaming, key management — is one endpoint you call from any machine, teammate, or agent you give a key to.

Quickstart

Pair a connector and register your first model end to end.

API keys

Create, scope, and rotate sk-tok-… keys for teammates and agents.

Compatible clients

39 clients confirmed working — coding agents, chat apps, SDKs.

Remote access comparison

How Tokios compares to Tailscale, Cloudflare Tunnel, and SSH tunnels.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Run vLLM locally and access it from anywhere through Tokios

Prerequisites

Step 1: Install and launch vLLM

Verify it’s running

Common launch configurations

Multi-GPU: tensor parallelism

Windows (WSL2)

Step 2: Test locally

Step 3: Install the Tokios connector

Step 4: Point the connector at vLLM

Step 5: Register a deployment

Step 6: Create an API key

Step 7: Connect from any client

Security model: outbound-only tunnel

FAQ

Your vLLM, accessible from anywhere

Quickstart

API keys

Compatible clients

Remote access comparison

​Prerequisites

​Step 1: Install and launch vLLM

​Verify it’s running

​Common launch configurations

​Multi-GPU: tensor parallelism

​Windows (WSL2)

​Step 2: Test locally

​Step 3: Install the Tokios connector

​Step 4: Point the connector at vLLM

​Step 5: Register a deployment

​Step 6: Create an API key

​Step 7: Connect from any client

​Security model: outbound-only tunnel

​FAQ

​Your vLLM, accessible from anywhere

Quickstart

API keys

Compatible clients

Remote access comparison

Prerequisites

Step 1: Install and launch vLLM

Verify it’s running

Common launch configurations

Multi-GPU: tensor parallelism

Windows (WSL2)

Step 2: Test locally

Step 3: Install the Tokios connector

Step 4: Point the connector at vLLM

Step 5: Register a deployment

Step 6: Create an API key

Step 7: Connect from any client

Security model: outbound-only tunnel

FAQ

Your vLLM, accessible from anywhere