Use a llama.cpp server from anywhere

By default, llama-server listens on http://127.0.0.1:8080/v1 — it isn’t reachable from another machine unless you bind it wider or forward the port. llama-server does ship an optional --api-key flag, but it’s off by default, and putting TLS in front of it is left entirely to you. Keep llama-server on localhost and run the Tokios connector next to it instead: the connector dials out to https://api.tokios.com, and your tools, teammates, or agents call one authenticated endpoint from anywhere.

`llama-server`’s auth is optional and off by default

llama-server is the OpenAI-compatible server that ships with llama.cpp, built to serve a GGUF model over HTTP. Run it and, unless you say otherwise, it accepts requests from anyone who can reach the port — no credential required:

llama-server -m model.gguf --port 8080

You can pass --api-key <key> to require a bearer token, and that’s a reasonable first layer. But there’s no default, and TLS termination is a separate, manual concern — llama-server doesn’t set that up for you.

This isn’t a knock on llama-server — it’s a lightweight, embeddable inference server, and leaving auth and TLS as opt-in flags keeps it simple to run on a trusted machine or behind your own infrastructure. Tokios is purpose-built for a different problem: turning that local endpoint into an authenticated API you can call from anywhere, without opening an inbound port or hand-rolling TLS. llama.cpp is a first-class Tokios backend.

The outbound-only way

Run llama-server locally, as usual

Keep llama-server on 127.0.0.1 — don’t bind it wider than loopback. Start it with your GGUF model:

llama-server -m model.gguf --port 8080

Leave it listening on the default http://127.0.0.1:8080.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing. Tokios shows a one-time claim code like TKS-XXXX-XXXX, valid for about 15 minutes.On the machine running llama-server, run the install one-liner for your OS:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Already have tokios-connector installed? Pass the code directly instead:

tokios-connector TKS-XXXX-XXXX

Back in the console, approve the device on the Setup or Connectors tab. The connector’s token is delivered over the tunnel once approved — you never see it on screen or paste it into a file.

Point the connector at llama-server's /v1 API

llama-server already speaks the OpenAI Chat Completions surface Tokios expects. For the basic single-model path, the console’s Setup wizard uses http://127.0.0.1:8080/v1 as the upstream automatically. If you’re serving multiple models through a tokios-connector.json config, set the upstream BaseUrl explicitly:

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8080/v1"
  }
}

See Connector config for the full multi-model file format.

In the Models tab, register the llama.cpp model and choose a public name — for example, llama-tunnel. That public name, not the GGUF file or its internal model id, is what clients send in the model field.

Create an API key

In the Keys tab, click Create key. Optionally scope it to specific deployments with model-name patterns (comma-separated globs, * for all). Copy the sk-tok-… key now — it’s shown only once.

Use it from anywhere

Once you have a deployment name and an API key, any OpenAI- or Anthropic-compatible client can reach your llama.cpp model — from another machine, a teammate’s laptop, or a hosted agent.

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

response = client.chat.completions.create(
    model="llama-tunnel",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model llama-tunnel

The model field is always your registered deployment name (llama-tunnel above), never the GGUF file name or a local model id. See Use Claude Code for persisting the environment variables across sessions.

FAQ

Does streaming work?

Yes. Set "stream": true and Tokios returns standard server-sent events (SSE) in OpenAI or Anthropic format, depending on which API surface you called.

I fine-tuned a GGUF model — can I serve that?

Yes. llama-server serves whatever GGUF file you point -m at, fine-tunes included. Register it as its own deployment with its own public name, the same way you’d register any other model. See Serve your fine-tune for the end-to-end path from a fine-tuned model to a Tokios deployment.

Can I serve more than one GGUF model?

Yes. Register additional deployments in the Models tab, or use a tokios-connector.json config file to define multiple upstreams and routes for a single connector — useful if you run more than one llama-server process, each on its own port. See Connector config.

Do I still need llama-server's --api-key flag?

You can keep it as defense in depth on the loopback connection between the connector and llama-server, but it isn’t what authenticates callers reaching you through Tokios. Client requests to https://api.tokios.com are authenticated with your sk-tok-… key; the connector is what’s allowed to reach 127.0.0.1:8080 at all.

Your model never leaves your machine — your endpoint works everywhere

llama-server keeps doing exactly what it does today, on localhost, with no inbound ports and no manual TLS to configure. The connector only dials out. Everything past that — auth, routing, streaming — is one endpoint you can call from any machine, teammate, or agent you choose to give a key to.

Quickstart

Pair a connector and register your first model end to end.

API keys

Create, scope, and rotate sk-tok-… keys for teammates and agents.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Use a llama.cpp server from anywhere — safely

`llama-server`’s auth is optional and off by default

The outbound-only way

Use it from anywhere

FAQ

Your model never leaves your machine — your endpoint works everywhere

Quickstart

API keys

​llama-server’s auth is optional and off by default

​The outbound-only way

​Use it from anywhere

​FAQ

​Your model never leaves your machine — your endpoint works everywhere

Quickstart

API keys

`llama-server`’s auth is optional and off by default

The outbound-only way

Use it from anywhere

FAQ

Your model never leaves your machine — your endpoint works everywhere