Self-hosted LLM — run it, serve it, control it

A self-hosted LLM (also called a private LLM or locallm) runs on hardware you own — your prompts, weights, and data never leave your network. Install a runtime (Ollama is the fastest start), pull an open-weight model, and you have a private AI in 2 commands. The step most guides skip: giving that model a secure, authenticated endpoint so your team, tools, and agents can call it from any device without exposing a port. Last verified: 2026-10-24

What is a self-hosted LLM?

A self-hosted LLM is an open-weight language model (Llama, Qwen, Mistral, DeepSeek, GLM) that you download and run on your own hardware using a serving runtime. Unlike a hosted API (OpenAI, Anthropic, Google), your prompts never leave your machine. You control the model, the data path, the uptime, and the access.

Why self-host?

Reason	What it means
Sovereignty	Your data never leaves your network. No third-party logging, no training on your prompts.
Control	Choose the model, quantization, context length, and serving config. No rate limits, no policy changes, no model deprecation.
Speed	Local inference eliminates network round-trip latency. Your model responds in milliseconds, not seconds.
Customization	Fine-tune on your own data for your specific domain. Cloud APIs don’t allow this — or charge a premium.
Compliance	Data residency, air-gap, and sovereignty requirements met by design. HIPAA, GDPR, and regulated industries.
Cost at scale	For high-volume use cases, local inference is cheaper than per-token cloud API billing. One hardware investment, unlimited inference.

Vitalik Buterin’s April 2026 post details the backdoor and censorship risks of hosted models and why local serving matters for sovereignty.

Pick your runtime

Runtime	Best for	Default port	OpenAI-compatible
Ollama	Easiest start, 1-command model pulls	11434	Yes (`/v1`)
LM Studio	GUI-first, beginner-friendly	1234	Yes (`/v1`)
llama.cpp	Max control, fine-grained quantization	8080	Yes (`/v1`)
vLLM	High-throughput multi-GPU serving	8000	Yes (`/v1`)

Start with Ollama. It handles model downloads, quantization, and GPU offloading automatically. Graduate to vLLM when you need concurrent-request throughput.

Hardware fit

Memory is the constraint. Formula: params × bytes-per-weight + overhead.

Model size	Q4 memory	Fits on
7–8B	~5 GB	8 GB GPU / 16 GB RAM
13–14B	~9 GB	16 GB GPU / 32 GB RAM
32–35B	~20 GB	24 GB GPU / 32 GB RAM
70B	~43 GB	48 GB GPU / 64 GB RAM
671B MoE	~350 GB+	Multi-GPU server

Example calc: 32B params × 0.5 bytes (4-bit) = 16 GB weights + ~4 GB KV cache/runtime = ~20 GB total. See the full hardware reference and model fit calculator for detailed sizing.

Run your first model

Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Pull a model

ollama pull qwen3:8b

Run it

ollama run qwen3:8b

You now have a private LLM. Your prompts never left your machine.

Ollama also serves an OpenAI-compatible API at http://127.0.0.1:11434/v1 — any OpenAI SDK can call it locally.

The missing layer: authenticated access

Here’s the wall every self-hosting guide stops at: your model works on localhost, but your laptop, teammate, coding agent, or n8n workflow can’t reach it securely.

Binding to 0.0.0.0 exposes the model with no authentication — anyone on the network can call it.
Port forwarding fails behind CGNAT (Starlink, apartment ISPs).
Tunnels (Cloudflare, ngrok) add buffering, timeouts, and request caps.

The Tokios connector adds the missing layer: an authenticated, HTTPS, OpenAI- and Anthropic-compatible endpoint — without opening a port.

Keep your model on localhost

Default binding. Don’t change it.

Pair the connector

tokios-connector TKS-XXXX-XXXX

Approve in the dashboard.

Give your model a public name (e.g., private-qwen3).

Create scoped API keys

1 key per teammate or tool. Scope by model-name glob. Revoke instantly.

Call from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "private-qwen3", "messages": [{"role": "user", "content": "Hello!"}]}'

You’ve got a private LLM running. Now give it a secure, authenticated endpoint your team and tools can call from anywhere.

Migration from cloud APIs

If you’re currently using OpenAI, Anthropic, or another cloud API, migration is straightforward:

Pick an equivalent open-weight model. Llama 3.1 70B, Qwen3 72B, or DeepSeek-R1 match GPT-4 on most benchmarks.
Serve it locally with Ollama or vLLM (see steps above).
Point your existing code at the local endpoint. Change base_url from https://api.openai.com/v1 to https://api.tokios.com/v1 (or your local http://127.0.0.1:PORT/v1).
Nothing else changes. The OpenAI SDK, Anthropic SDK, and all compatible clients work identically.

Your app code doesn’t change. Your prompts don’t change. Your workflow doesn’t change. The only difference: your data stays on your network. See Bring your AI in-house for the full migration walkthrough.

FAQ

What is a private LLM?

A language model that runs on hardware you control. Your prompts, outputs, and data never leave your network. Open-weight models (Llama, Qwen, Mistral, DeepSeek) make this possible — you download the weights and serve them yourself.

Can I self-host ChatGPT?

Not ChatGPT itself (it’s proprietary), but you can self-host models with comparable quality — Llama 3.1 70B, Qwen3 72B, DeepSeek-R1 70B — and get a ChatGPT-like experience with Open WebUI or any OpenAI-compatible client.

How much does it cost to self-host an LLM?

Hardware: a 16 GB GPU (RTX 4080, ~$1,200) runs 8B models comfortably. A 24 GB RTX 4090 (~$1,600) handles 32B. After hardware, the marginal cost per token is electricity only. No subscription, no per-token billing.

Is self-hosting an LLM safe?

Safer than hosted APIs for data privacy — nothing leaves your network. The risk is exposing the serving port without auth. Keep the model on localhost and use an authenticated endpoint (like Tokios) for remote access.

Can I run a private LLM offline / air-gapped?

Yes. Ollama and llama.cpp work fully offline once the model is downloaded. For air-gapped environments, download the model file on a connected machine and transfer it. The Tokios connector requires outbound internet to reach api.tokios.com — for fully air-gapped use, call the model’s localhost API directly.

How do I give my team access to my self-hosted model?

Create a separate sk-tok-… API key per teammate in the Tokios dashboard. Scope each key to specific deployments. Revoke access instantly without touching the model or server. No shared passwords, no exposed ports. See Team access to local models.

Team access

Full walkthrough for giving your team API access to your local models.

Bring your AI in-house

Migration guide for teams switching from cloud APIs to local inference.

Ollama remote access

The full guide to authenticated remote access for Ollama.

Quickstart

Pair a connector and register your first model.

Model fit guide

Find the right model for your exact hardware.

What is a local LLM?

Complete guide to running large language models locally.

Getting started with local LLMs

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Self-hosted LLM — run it, serve it, control it

What is a self-hosted LLM?

Why self-host?

Pick your runtime

Hardware fit

Run your first model

The missing layer: authenticated access

Migration from cloud APIs

FAQ

Team access

Bring your AI in-house

Ollama remote access

Quickstart

Model fit guide

What is a local LLM?

​What is a self-hosted LLM?

​Why self-host?

​Pick your runtime

​Hardware fit

​Run your first model

​The missing layer: authenticated access

​Migration from cloud APIs

​FAQ

Team access

Bring your AI in-house

Ollama remote access

Quickstart

Model fit guide

What is a local LLM?

What is a self-hosted LLM?

Why self-host?

Pick your runtime

Hardware fit

Run your first model

The missing layer: authenticated access

Migration from cloud APIs

FAQ