Use an Xinference server from anywhere

Xinference (Xorbits Inference) serves models behind an OpenAI-compatible API on your local machine — by default that API only answers on localhost, so it isn’t reachable from another machine unless you bind it wider or forward a port. Keep Xinference on localhost and run the Tokios connector next to it instead: the connector dials out to https://api.tokios.com, and your tools, teammates, or agents call one authenticated endpoint from anywhere.

Advanced runtime. Xinference is a model-serving framework aimed at more involved setups. If you just want the easiest way to serve a model, start with Ollama or LM Studio.

Xinference exposes an OpenAI-compatible Chat Completions API, so it works with Tokios like any OpenAI-compatible server: the connector forwards to whatever /v1 BaseUrl you set, regardless of which server implements that surface. Ollama, llama.cpp, vLLM, and LM Studio are the backends Tokios documents most closely, but anything OpenAI-compatible — including Xinference — works the same way.

The outbound-only way

Run Xinference locally, as usual

Start the local Xinference server — for example with xinference-local — and launch a model through its UI or CLI. The exact startup command and flags can vary by version, so check yours.

xinference-local

Xinference commonly serves its OpenAI-compatible API on port 9997 — confirm the default port for your install. Keep it bound to 127.0.0.1 rather than 0.0.0.0, and don’t forward the port on your router.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing. Tokios shows a one-time claim code like TKS-XXXX-XXXX, valid for about 15 minutes.On the machine running Xinference, run the install one-liner for your OS:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Already have tokios-connector installed? Pass the code directly instead:

tokios-connector TKS-XXXX-XXXX

Back in the console, approve the device on the Setup or Connectors tab. The connector’s token is delivered over the tunnel once approved — you never see it on screen or paste it into a file.

Point the connector at Xinference's /v1 API

For the basic single-model path, the console’s Setup wizard asks for your upstream base URL — enter Xinference’s OpenAI-compatible endpoint, http://127.0.0.1:<port>/v1 (confirm the port against your install). If you’re serving multiple models through a tokios-connector.json config, set the upstream BaseUrl explicitly:

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:9997/v1"
  }
}

Confirm 9997 against your running Xinference instance before using it. See Connector config for the full multi-model file format.

In the Models tab, register the Xinference model and choose a public name — for example, xinference-tunnel. That public name, not the model id you launched in Xinference, is what clients send in the model field.

Create an API key

In the Keys tab, click Create key. Optionally scope it to specific deployments with model-name patterns (comma-separated globs, * for all). Copy the sk-tok-… key now — it’s shown only once.

Use it from anywhere

Once you have a deployment name and an API key, any OpenAI- or Anthropic-compatible client can reach your Xinference model — from another machine, a teammate’s laptop, or a hosted agent.

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "xinference-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="xinference-tunnel",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model xinference-tunnel

The model field is always your registered deployment name (xinference-tunnel above), never the model id you launched in Xinference. See Use Claude Code for persisting the environment variables across sessions.

FAQ

Is Xinference officially supported?

Ollama, llama.cpp, vLLM, and LM Studio are the backends Tokios documents most closely. But the connector works against any server exposing an OpenAI-compatible Chat Completions API at a /v1 base URL, and Xinference does that, so it works as a generic OpenAI-compatible upstream the same way. Confirm behavior against your own Xinference version before relying on it in production.

Does streaming work?

Yes, if the underlying Xinference model and server support streaming responses. Set "stream": true and Tokios returns standard server-sent events (SSE) in OpenAI or Anthropic format, depending on which API surface you called.

Can I serve more than one Xinference model?

Yes. Register additional deployments in the Models tab, or use a tokios-connector.json config file to define multiple upstreams and routes for a single connector — useful if you launch more than one model in Xinference. See Connector config.

Your model never leaves your machine — your endpoint works everywhere

Xinference keeps doing exactly what it does today, on localhost, with no inbound ports and no endpoints exposed beyond /v1. The connector only dials out. Everything past that — auth, routing, streaming — is one endpoint you can call from any machine, teammate, or agent you choose to give a key to.

Supported backends

See which local runtimes Tokios explicitly tests and supports today.

API keys

Create, scope, and rotate sk-tok-… keys for teammates and agents.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Use an Xinference server from anywhere — safely

The outbound-only way

Use it from anywhere

FAQ

Your model never leaves your machine — your endpoint works everywhere

Supported backends

API keys

​The outbound-only way

​Use it from anywhere

​FAQ

​Your model never leaves your machine — your endpoint works everywhere

Supported backends

API keys

The outbound-only way

Use it from anywhere

FAQ

Your model never leaves your machine — your endpoint works everywhere