Run vLLM on Windows with WSL2 and NVIDIA GPUs

vLLM does not run natively on Windows. The supported path is WSL2 with Docker and the NVIDIA Container Toolkit, which gives you full GPU passthrough and access to the entire Linux inference stack without dual-booting. This guide walks through the complete setup on Windows 11 with NVIDIA GPUs.

Last tested: July 2026. A community PR for native Windows support was submitted but has not been merged into a stable release. Until then, WSL2 + Docker remains the recommended path.

Prerequisites

Windows 11 (22H2 or later) with WSL2 enabled
1 or more NVIDIA GPUs with driver version ≥535
Docker Desktop for Windows with the WSL2 backend enabled
At least 16 GB of system RAM (32 GB recommended for models ≥13B)

Step 1: install WSL2 with GPU support

Open PowerShell as Administrator and install WSL2:

wsl --install

This installs Ubuntu by default. After reboot, confirm the WSL kernel version:

uname -r

You need kernel ≥5.15 for GPU passthrough. Update if necessary:

sudo apt update && sudo apt upgrade -y

Verify that your GPU is visible inside WSL2:

nvidia-smi

You should see your GPU listed with the correct driver version. If nvidia-smi reports no devices, update your Windows NVIDIA driver to ≥535 and restart WSL:

wsl --shutdown

Step 2: install Docker with NVIDIA Container Toolkit

Install Docker Desktop on Windows.
Open Docker Desktop settings and confirm Use the WSL 2 based engine is checked.
Under Resources → WSL Integration, enable your Ubuntu distribution.

Inside WSL2, install the NVIDIA Container Toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit

Restart Docker after installation:

sudo service docker restart

Step 3: launch vLLM with Docker Compose

Create a docker-compose.yml file:

docker-compose.yml

services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --model Qwen/Qwen2.5-7B-Instruct
      --gpu-memory-utilization 0.9
      --max-model-len 8192

Start the container:

docker compose up -d

Watch the logs while the model downloads and loads:

docker compose logs -f vllm

When you see Uvicorn running on http://0.0.0.0:8000, the server is ready.

Step 4: verify it works

Send a test request from inside WSL2:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Expected response: a JSON object with a choices array containing the model’s reply.

Multi-GPU configuration (MCDM)

WSL2 supports multi-GPU tensor parallelism through the Microsoft Compute Driver Model (MCDM). This replaces the older TCC mode and requires NVIDIA driver ≥535.

Switch to MCDM mode

On the Windows host, open an Administrator command prompt:

nvidia-smi -g 0 -dm 0

Repeat for each GPU (replace 0 with the GPU index), then reboot. After reboot, confirm all GPUs appear in WSL2:

nvidia-smi

Launch with tensor parallelism

Update your docker-compose.yml command to use multiple GPUs:

    command: >
      --model Qwen/Qwen2.5-72B-Instruct
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.9
      --max-model-len 4096

Tensor parallelism splits layers across GPUs and is preferred over pipeline parallelism for single-node multi-GPU setups. It delivers better throughput for the decode phase.

Connect from the Windows host

WSL2 runs in a lightweight VM with its own network. Docker’s port mapping forwards localhost:8000 from the WSL2 VM to the Windows host automatically, so you can reach vLLM from Windows applications:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"Qwen/Qwen2.5-7B-Instruct\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]}"

If localhost forwarding doesn’t work, find the WSL2 IP address and use it directly:

hostname -I

Then on Windows:

curl http://<WSL2_IP>:8000/v1/chat/completions ...

WSL2’s IP address changes on every restart. If you rely on the IP directly, you need to update your configuration after each reboot. Prefer localhost forwarding when possible.

Expose your endpoint with Tokios

Running vLLM on WSL2 gives you local inference, but it’s only reachable from the same machine. Install the Tokios connector inside WSL2 alongside your vLLM container to expose your model through a secure tunnel — accessible from any device, without port forwarding or firewall rules:

# Inside WSL2, install and pair the connector
curl -fsSL https://install.tokios.com | bash
tokios-connector pair

Once paired, register your vLLM deployment on the Models tab and generate an API key. Your model is now reachable at https://api.tokios.com/v1 from anywhere. See Connector install for the full walkthrough.

Troubleshooting

GPU not detected in WSL2

Your NVIDIA driver is too old or WSL2 kernel is outdated. Update the Windows NVIDIA driver to ≥535, then run wsl --update from PowerShell and restart WSL with wsl --shutdown.

CUDA kernel image unavailable

Specific vLLM versions (v0.22/v0.23) ship CUDA kernels that don’t match every WSL2 CUDA toolkit version. Pin your Docker image to a known-good tag, for example vllm/vllm-openai:v0.21.0, instead of latest.

Out of memory on multi-GPU

vLLM reserves more VRAM for CUDA graphs than Ollama. If a 32B model OOMs on 2x RTX 3090, lower --gpu-memory-utilization to 0.85 or reduce --max-model-len.

Docker container exits immediately

Check that Docker Desktop’s WSL2 integration is enabled for your distribution. Run docker compose logs vllm to see the exit reason — it’s usually a missing GPU driver or an incorrect model path.

MCDM not enabled

If nvidia-smi inside WSL2 only shows 1 GPU on a multi-GPU system, MCDM may not be active. Run nvidia-smi -g <GPU_ID> -dm 0 on the Windows host for each GPU, then reboot.

Native Windows support status

A community fork by SystemPanic adds native Windows support to vLLM without WSL2 or Docker. The PR was submitted in early 2025 but has not been merged into a stable release. Track progress on the vLLM GitHub repository. Until native support ships in a stable release, WSL2 + Docker is the production path for Windows.

Next steps

vLLM remote access

Expose your vLLM server to any device with Tokios — no port forwarding required.

Model fit

Find which models fit your GPU’s VRAM and what throughput to expect.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Run vLLM on Windows with WSL2 and NVIDIA GPUs

Prerequisites

Step 1: install WSL2 with GPU support

Step 2: install Docker with NVIDIA Container Toolkit

Step 3: launch vLLM with Docker Compose

Step 4: verify it works

Multi-GPU configuration (MCDM)

Switch to MCDM mode

Launch with tensor parallelism

Connect from the Windows host

Expose your endpoint with Tokios

Troubleshooting

Native Windows support status

Next steps

vLLM remote access

Model fit

​Prerequisites

​Step 1: install WSL2 with GPU support

​Step 2: install Docker with NVIDIA Container Toolkit

​Step 3: launch vLLM with Docker Compose

​Step 4: verify it works

​Multi-GPU configuration (MCDM)

​Switch to MCDM mode

​Launch with tensor parallelism

​Connect from the Windows host

​Expose your endpoint with Tokios

​Troubleshooting

​Native Windows support status

​Next steps

vLLM remote access

Model fit

Prerequisites

Step 1: install WSL2 with GPU support

Step 2: install Docker with NVIDIA Container Toolkit

Step 3: launch vLLM with Docker Compose

Step 4: verify it works

Multi-GPU configuration (MCDM)

Switch to MCDM mode

Launch with tensor parallelism

Connect from the Windows host

Expose your endpoint with Tokios

Troubleshooting

Native Windows support status

Next steps