Run gpt-oss locally and use it from anywhere

gpt-oss is OpenAI’s open-weight reasoning model family. It gives you two mixture-of-experts sizes: gpt-oss-20b for local and lower-latency use, and gpt-oss-120b for workstation or server hardware. Run the model locally first, then use the Tokios connector to expose that local runtime through https://api.tokios.com without opening inbound ports.

Run it locally with Ollama

Start with 20b unless you already know your machine can hold the larger model:

ollama run gpt-oss:20b

For the larger tier:

ollama run gpt-oss:120b

Leave Ollama on its default loopback address, http://127.0.0.1:11434. Tokios will connect to Ollama’s OpenAI-compatible /v1 API later.

Variants and tags

Tags and quantizations change over time. Confirm the exact tag on Ollama’s gpt-oss library page before you standardize on it.

Variant	Ollama tag	Hugging Face repo	Parameters	Active parameters	Download size	Context	Input	License
gpt-oss 20B	`gpt-oss:20b`	`openai/gpt-oss-20b`	21B MoE	3.6B per token	14 GB	128K	Text	Apache 2.0
gpt-oss 120B	`gpt-oss:120b`	`openai/gpt-oss-120b`	117B MoE	5.1B per token	65 GB	128K	Text	Apache 2.0

Sources: Ollama gpt-oss tags, OpenAI gpt-oss announcement, openai/gpt-oss-20b, and openai/gpt-oss-120b.

The Ollama gpt-oss:latest tag currently points at the 20b download. Pin gpt-oss:20b or gpt-oss:120b in production so the local upstream model id stays clear.

Quantization and formats

OpenAI ships gpt-oss in MXFP4: the MoE weights are quantized to 4.25 bits per parameter. Ollama supports that format natively for gpt-oss, so the beginner path does not need a manual conversion step. OpenAI states that gpt-oss-20b can run within 16 GB of memory and gpt-oss-120b can fit on a single 80 GB GPU when using its native quantization. Use the format, not just the parameter count, when you choose hardware:

Format	Where to use it	Notes
`MXFP4`	Ollama, llama.cpp, vLLM	Native gpt-oss format. This is the main local path.
`bf16` / `FP16`	Research or high-memory serving	Much larger than the local-friendly quantized path.
`GGUF`	llama.cpp-based apps when a compatible build exists	Check the model card or runtime catalog before you pull a third-party conversion.
`AWQ` / `GPTQ` / `NVFP4`	Only if your serving repo publishes that format	Do not assume a quant exists just because another model family has it.

See Quantization formats and the hardware they need for the format matrix, and what quantization is for the Q4 / Q8 naming basics.

Hardware fit

Treat this as a planning guide for the native MXFP4 / Ollama path at a modest context length. The real limit is the downloaded model size plus KV cache for your context.

Target	`gpt-oss:20b`	`gpt-oss:120b`
Laptop CPU or 8 GB GPU	Won’t fit comfortably	Won’t fit
16 GB GPU or 16 GB unified memory	Tight; keep context modest	Won’t fit
RTX 3090 / 4090, 24 GB	Comfortable	Won’t fit
RTX 5090, 32 GB	Comfortable, more room for context	Won’t fit
RTX 6000 Ada / A6000, 48 GB	Comfortable	Won’t fit
Apple Silicon or Strix Halo, 96-128 GB unified	Comfortable	Fits by capacity; expect lower bandwidth than dedicated VRAM
Single 80 GB datacenter GPU	Comfortable	Fits by OpenAI’s target
Multi-GPU workstation or server	Comfortable	Comfortable

For the sizing method, see What size model fits your GPU?. For the prompt lifecycle and why long context adds memory, see KV cache and context.

Run it: beginner to advanced

Ollama (easiest)
LM Studio (GUI)
Unsloth Studio (no-code)
vLLM (advanced)
SGLang (advanced)

ollama run gpt-oss:20b

Use gpt-oss:120b only on hardware with enough memory. Ollama publishes both tags and supports the model’s native tool and reasoning features through its chat API.

Search for gpt-oss in LM Studio’s model catalog, choose the 20b or 120b build that fits your memory, and start the local server from the Developer tab. LM Studio’s OpenAI-compatible API normally uses:

http://127.0.0.1:1234/v1

Use that URL as the Tokios upstream instead of Ollama’s 11434 URL if LM Studio is the runtime.

Use Unsloth Studio when you want a GUI workflow around local open models, fine-tuning, or export. If your Studio build exposes an OpenAI-compatible inference endpoint, point Tokios at that /v1 URL. Otherwise export or load the model through Ollama, LM Studio, or llama.cpp and connect Tokios to that runtime.

Hugging Face lists a direct vLLM serve command for both gpt-oss repos:

vllm serve openai/gpt-oss-20b

For 120b, swap the model id:

vllm serve openai/gpt-oss-120b

vLLM exposes an OpenAI-compatible API on http://127.0.0.1:8000/v1 by default.

Hugging Face lists this SGLang launch shape:

python3 -m sglang.launch_server \
  --model-path openai/gpt-oss-20b \
  --host 127.0.0.1 \
  --port 30000

For 120b, use --model-path openai/gpt-oss-120b. Point Tokios at http://127.0.0.1:30000/v1.

Capabilities

Reasoning: gpt-oss supports configurable reasoning effort: low, medium, and high. Set the effort through the runtime’s supported prompt or parameter mechanism.
Tool use: gpt-oss was trained for the harmony response format and supports native function calling, web browsing, Python tool calls, and structured outputs. Use a runtime that applies the harmony format for you.
Context: Both sizes support up to 128K context. Raise your runtime context setting if the default is lower.
Vision: gpt-oss is text-only.
Fine-tuning: OpenAI describes the family as fine-tunable. Use your runtime or training tool’s current gpt-oss support before planning a fine-tune.

gpt-oss should be used with OpenAI’s harmony response format. Ollama, Transformers chat templates, and other supported runtimes can apply it for you. If you change runtimes, verify harmony support before you rely on reasoning traces or tool calls.

Connect it with Tokios

Keep your local runtime running

Start gpt-oss in Ollama, LM Studio, vLLM, SGLang, or another OpenAI-compatible runtime. Keep the runtime bound to loopback.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing. Tokios shows a one-time claim code like TKS-XXXX-XXXX.Run the installer on the machine that runs the model:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Already have tokios-connector installed? Pass the claim code directly:

tokios-connector TKS-XXXX-XXXX

Back in the dashboard, approve the device on the Setup or Connectors tab.

Point the connector at the runtime's /v1 API

For Ollama, use:

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:11434/v1"
  }
}

For LM Studio, use http://127.0.0.1:1234/v1. For vLLM, use http://127.0.0.1:8000/v1. For SGLang, use the port you launched, such as http://127.0.0.1:30000/v1.

In the Models tab, register the upstream model and choose a public name, for example gpt-oss-tunnel. API clients send this deployment name, not gpt-oss:20b or openai/gpt-oss-20b, in the model field.

Create an API key

In the Keys tab, click Create key. Copy the sk-tok-... API key when it appears.

Use it from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Explain why long context uses more memory."}]
  }'

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model gpt-oss-tunnel

The model field is always your registered deployment name. Tokios forwards requests to your local runtime; it does not change gpt-oss’s reasoning, context, or tool-calling behavior.

Caveats

gpt-oss:120b is a different hardware tier from gpt-oss:20b. Use 20b first unless you have 80 GB-class GPU memory or large unified memory.
The model is text-only. Pick a multimodal model if your client needs images.
Tool calling depends on the runtime applying the harmony format correctly.
Long context increases KV-cache memory. Test your real workload before promising a 128K deployment to teammates.

Quickstart

Pair a connector and register your first deployment end to end.

GPU-to-model fit

Match gpt-oss sizes to your GPU or unified memory.

Choose a model by task

Compare gpt-oss against other local models for chat, coding, and agent workloads.

Quantization and hardware

Check which formats run on which hardware.

Get Started

Models

Concepts

Reference

Run gpt-oss locally and use it from anywhere

Run it locally with Ollama

Variants and tags

Quantization and formats

Hardware fit

Run it: beginner to advanced

Capabilities

Connect it with Tokios

Use it from anywhere

Caveats

Quickstart

GPU-to-model fit

Choose a model by task

Quantization and hardware

​Run it locally with Ollama

​Variants and tags

​Quantization and formats

​Hardware fit

​Run it: beginner to advanced

​Capabilities

​Connect it with Tokios

​Use it from anywhere

​Caveats

Quickstart

GPU-to-model fit

Choose a model by task

Quantization and hardware

Run it locally with Ollama

Variants and tags

Quantization and formats

Hardware fit

Run it: beginner to advanced

Capabilities

Connect it with Tokios

Use it from anywhere

Caveats