Hardening a self-hosted LLM: a checklist

Running a model locally with Ollama, llama.cpp, vLLM, or LM Studio is the easy part. Making that endpoint reachable by a teammate, an agent, or a second machine — without turning it into an open door — is where most setups go wrong. This checklist walks through the items that matter, in the order they tend to bite people, and notes which ones stop being your problem at all once you put an outbound-only connector like Tokios in front. This is a checklist, not a guarantee. Following it reduces exposure; it doesn’t eliminate risk, and nothing here is a claim that any specific setup — Tokios included — is unbreakable.

​

1. Bind address: keep it on loopback

Every runtime in this space defaults to 127.0.0.1 (loopback-only) for a reason: it’s the single control that keeps the API from being reachable by anything other than the machine it’s running on.

• Ollama listens on 127.0.0.1:11434 by default. Setting OLLAMA_HOST=0.0.0.0 makes it listen on every network interface — reachable from your LAN and, if the port is also forwarded, the open internet.
• llama.cpp server, vLLM, and LM Studio have the same pattern: a loopback default, with a flag or setting to bind to 0.0.0.0 or a specific LAN interface.

Checklist item: if you don’t have a specific reason to bind wider than loopback, don’t. If you do need LAN or remote access, that’s what the rest of this checklist — or an outbound-only connector — is for.

​

2. Authentication: most runtimes have none, or optional

Binding wider than loopback without authentication means anything that can reach the port can call the API with no credential — list models, run inference, pull models. Checklist item: if more than one person or service calls your endpoint, a single shared static key (or no key) can’t tell callers apart and can’t be revoked for just one of them. See Add API keys to a keyless local model for the tradeoffs in more depth.

​

3. Firewall and inbound ports

If you do bind wider than loopback, the next control is the firewall in front of it:

• Restrict the inbound port (11434 for Ollama, 8080 for llama.cpp, 8000 for vLLM, 1234 for LM Studio) to known source IPs where possible, rather than leaving it open to any address.
• Remember that forwarding a port on your router only works if you have a stable public IP. Many residential and mobile connections sit behind CGNAT, where there’s no public IP to forward to in the first place — see Reach a model behind CGNAT.
• Internet-wide scanners (Shodan, Censys) index newly opened services on the order of hours to days. An open, unauthenticated port doesn’t stay unnoticed for long.

Checklist item: treat “the port is only forwarded temporarily” as still discoverable. If a port is open, assume it will be found.

​

4. Reverse-proxy pitfalls

Fronting a keyless runtime with nginx or Caddy is the most common DIY fix for the auth gap above, and it introduces its own list of things to get right:

• TLS termination. The proxy needs a valid certificate; a self-signed one will break most HTTP clients’ default trust checks.
• Streaming responses. LLM APIs stream tokens over Server-Sent Events (SSE). A reverse proxy (or an upstream CDN) that buffers responses instead of forwarding them as they arrive can turn a chat that should stream token-by-token into one that hangs until the whole response is done, or times out first.
• Auth logic lives in your proxy config, not the model server. You are now responsible for keeping the proxy’s auth rules correct, in sync, and tested — the runtime itself still has no concept of per-user keys.
• Timeouts. Long generations can exceed a proxy’s or edge network’s default request timeout, cutting the response off mid-stream.

Checklist item: if you stand up a reverse proxy, explicitly test a slow streaming response end to end — not just a fast, short one — before you trust it in production.

​

5. Keep the runtime patched

Local inference servers are software with a normal vulnerability lifecycle, and named CVEs in this space do get exploited once they’re public. Ollama, for example, has had security advisories in its release history — check the project’s own GitHub security advisories or the NVD for the current list before relying on a specific CVE ID or fixed-in version. Checklist item: subscribe to your runtime’s release notes or security advisories, and don’t leave a version running indefinitely just because it works. This applies whether the server is on loopback or exposed — a compromised host is a problem even if nothing else about your network is wrong.

​

What outbound-only removes from this list entirely

Every item above exists because something has to listen for inbound connections in order to be reachable. An outbound-only connector — the Tokios connector is one example — flips that: the connector process runs next to your model and dials out to https://api.tokios.com, so nothing on your machine or network ever listens on a port for the model traffic. That removes some checklist items outright, and leaves others unchanged:

• No inbound port to bind or forward. The model stays on 127.0.0.1; the connector reaches out, so there’s no 0.0.0.0 bind, no port-forward rule, and no CGNAT problem to solve for reachability.
• No firewall rule for the model port. There’s nothing inbound for a firewall to allow or block on the model’s behalf.
• No reverse proxy needed for TLS or routing. The connection to https://api.tokios.com is already TLS; you aren’t running or maintaining nginx or Caddy to get there.
• Real, scoped API keys instead of one shared static key. Every request authenticates with a Tokios sk-tok-… key that can be scoped to specific model deployments (comma-separated globs like gemma*, gpt-4*) and revoked individually — so removing one teammate’s or service’s access doesn’t require rotating a key everyone else uses too. See Add API keys to a keyless local model.
• What doesn’t change: patching. Tokios sits in front of your runtime; it doesn’t patch it. Keeping Ollama, llama.cpp, vLLM, or LM Studio itself up to date is still on you, connector or not.

If a request does hit an error, check Troubleshooting for the common causes before assuming it’s a security issue — most connection failures are configuration, not compromise.

​

Graduate to an outbound-only setup