Skip to main content
The Tokios Connector forwards to any model server that exposes an OpenAI-compatible Chat Completions API (/v1/chat/completions) on your machine. Ollama, llama.cpp, vLLM, LM Studio, MLX Studio, and MLX-LM Server all do. The gateway translates each incoming request to Chat Completions and sends it down the tunnel; the connector forwards it to the upstream BaseUrl you set in tokios-connector.json. Point the connector at your server by setting that BaseUrl — including the /v1 segment.
Always include /v1 in BaseUrl, and use your backend’s OpenAI-compatible endpoint. For Ollama that’s the /v1 API (not the native /api).
New to local models? Start with the easiest to run: Ollama (one command) or LM Studio (a point-and-click desktop app). vLLM and SGLang are powerful GPU servers but need more setup — reach for them once you’re comfortable and want higher throughput.

​
Ease of setup

Whichever you pick, Tokios connects to it the same way — set the upstream BaseUrl (with /v1) and you get one endpoint.
This list isn’t exclusive. The connector forwards to any server that exposes an OpenAI-compatible Chat Completions API at a /v1 base URL — so SGLang, Xinference, LocalAI, and similar all work too. Ollama, llama.cpp, vLLM, and LM Studio are simply the ones documented most closely here.
Jan can also run as a client — instead of serving a model locally, it can call a bigger model running elsewhere through Tokios. See Use Jan with a model on your own hardware.

​
Default ports

Drop the matching Upstream into your tokios-connector.json (see Connector Config for the full file):
The model id your server uses (for Ollama, whatever you ollama run) is what you list in the connector’s Models / Routes. You give it a public API name when you register a deployment.

​
Backend details

​
Ollama

Upstream: http://127.0.0.1:11434/v1 Ollama serves an OpenAI-compatible API at /v1 on port 11434. Pull and run your model first — the id you give Ollama is the upstream model id the connector serves:

​
llama.cpp

Upstream: http://127.0.0.1:8080/v1 Run the bundled OpenAI-compatible server, llama-server. It listens on port 8080 by default and serves /v1:

​
vLLM

Upstream: http://127.0.0.1:8000/v1 vLLM’s OpenAI-compatible server listens on port 8000. Start it with:

​
LM Studio

Upstream: http://127.0.0.1:1234/v1 LM Studio includes a built-in OpenAI-compatible server:
  1. Open LM Studio and go to the Developer (Local Server) tab.
  2. Load a model and start the server. It listens on port 1234.

​
MLX Studio

Upstream: http://127.0.0.1:8000/v1 MLX Studio is a native macOS app (macOS 14.0+ Sonoma, all Apple Silicon M1–M4) that wraps the vMLX inference server. It ships with a bundled Python 3.12 environment — no terminal, no pip, no venv required. Download the app, pick a model from any mlx-community repository on Hugging Face, and start the built-in API server. MLX Studio includes chat, vision, image generation, text-to-speech, speech-to-text, and tool calling. Its inference engine supports JANG mixed-precision quantization, continuous batching with PagedAttention, speculative decoding, KV cache quantization, and prefix caching. An interactive API documentation and testing playground is built in, and a menu-bar mode gives quick access without a full window.
  1. Open MLX Studio and download a model from the Model Manager.
  2. Start the API server from the Server tab. It listens on port 8000.
MLX Studio is the easiest GUI path for Mac users who want a native Apple Silicon experience without touching the terminal. For a headless or scriptable setup on the same hardware, use MLX-LM Server instead.

​
Lemonade Server

Upstream: http://127.0.0.1:13305/v1 Lemonade Server is an open-source inference server backed by AMD. It auto-detects your hardware and selects the best backend — Vulkan, ROCm, CUDA, or Metal — so you don’t pick a backend yourself. It is the only backend here that supports AMD Ryzen AI NPUs. Install via MSI on Windows, APT PPA on Ubuntu, or AUR on Arch. See lemonade-server.ai for details. Lemonade Server exposes an OpenAI-compatible API at /api/v1 on port 13305 by default. It also supports Ollama-compatible and Anthropic-compatible endpoints.
Lemonade Server simplifies setup on AMD Strix Halo and DGX Spark boxes — it picks Vulkan or CUDA automatically, no manual backend flags needed.

​
MLX-LM Server

Upstream: http://127.0.0.1:8080/v1 MLX-LM Server is part of the mlx-lm Python package and runs natively on Apple Silicon. It uses the MLX framework’s unified-memory architecture with zero-copy data paths, so model weights live in the same memory pool as the CPU and GPU — no separate VRAM to manage. Install the package and start the server with any mlx-community model from Hugging Face:
The server listens on port 8080 by default and exposes an OpenAI-compatible Chat Completions API at /v1. It supports continuous batching, tool calling, structured output, and prompt caching. Available quantizations include 4-bit (default), 8-bit, and BF16.
MLX-LM Server is the native Apple Silicon path — it competes with or slightly outperforms llama.cpp on M-series chips and takes full advantage of M5 Neural Accelerators (4x faster prompt processing). See mlx-framework.org for details.
Running on a different port? Just change the port in BaseUrl. Any server that speaks OpenAI Chat Completions works.