Skip to main content
The Tokios Connector forwards to any model server that exposes an OpenAI-compatible Chat Completions API (/v1/chat/completions) on your machine. Ollama, llama.cpp, vLLM, and LM Studio all do. The gateway translates each incoming request to Chat Completions and sends it down the tunnel; the connector forwards it to the upstream BaseUrl you set in tokios-connector.json. Point the connector at your server by setting that BaseUrl — including the /v1 segment.
Always include /v1 in BaseUrl, and use your backend’s OpenAI-compatible endpoint. For Ollama that’s the /v1 API (not the native /api).
New to local models? Start with the easiest to run: Ollama (one command) or LM Studio (a point-and-click desktop app). vLLM and SGLang are powerful GPU servers but need more setup — reach for them once you’re comfortable and want higher throughput.

​
Ease of setup

Whichever you pick, Tokios connects to it the same way — set the upstream BaseUrl (with /v1) and you get one endpoint.
This list isn’t exclusive. The connector forwards to any server that exposes an OpenAI-compatible Chat Completions API at a /v1 base URL — so SGLang, Xinference, TGI, LocalAI, and similar all work too. Ollama, llama.cpp, vLLM, and LM Studio are simply the ones documented most closely here.
Jan can also run as a client — instead of serving a model locally, it can call a bigger model running elsewhere through Tokios. See Use Jan with a model on your own hardware.

​
Default ports

Drop the matching Upstream into your tokios-connector.json (see Connector Config for the full file):
The model id your server uses (for Ollama, whatever you ollama run) is what you list in the connector’s Models / Routes. You give it a public API name when you register a deployment.

​
Backend details

​
Ollama

Upstream: http://127.0.0.1:11434/v1 Ollama serves an OpenAI-compatible API at /v1 on port 11434. Pull and run your model first — the id you give Ollama is the upstream model id the connector serves:

​
llama.cpp

Upstream: http://127.0.0.1:8080/v1 Run the bundled OpenAI-compatible server, llama-server. It listens on port 8080 by default and serves /v1:

​
vLLM

Upstream: http://127.0.0.1:8000/v1 vLLM’s OpenAI-compatible server listens on port 8000. Start it with:

​
LM Studio

Upstream: http://127.0.0.1:1234/v1 LM Studio includes a built-in OpenAI-compatible server:
  1. Open LM Studio and go to the Developer (Local Server) tab.
  2. Load a model and start the server. It listens on port 1234.
Running on a different port? Just change the port in BaseUrl. Any server that speaks OpenAI Chat Completions works.