> ## Documentation Index > Fetch the complete documentation index at: https://docs.tokios.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel. > Use 'connector' for the local tokios-connector process. > Use 'deployment' for the public model name in the API model field. > Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend. > API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-. # Self-hosted LLM — run it, serve it, control it > How to self-host a private LLM (locallm) on your own hardware, pick the right runtime and model, and give it a secure authenticated endpoint your team can call from anywhere. A self-hosted LLM (also called a **private LLM** or **locallm**) runs on hardware you own — your prompts, weights, and data never leave your network. Install a runtime (Ollama is the fastest start), pull an open-weight model, and you have a private AI in 2 commands. The step most guides skip: giving that model a **secure, authenticated endpoint** so your team, tools, and agents can call it from any device without exposing a port. Last verified: 2026-10-24 ## What is a self-hosted LLM? A self-hosted LLM is an open-weight language model (Llama, Qwen, Mistral, DeepSeek, GLM) that you download and run on your own hardware using a serving runtime. Unlike a hosted API (OpenAI, Anthropic, Google), your prompts never leave your machine. You control the model, the data path, the uptime, and the access. ## Why self-host? | Reason | What it means | | - | - | | **Sovereignty** | Your data never leaves your network. No third-party logging, no training on your prompts. | | **Control** | Choose the model, quantization, context length, and serving config. No rate limits, no policy changes, no model deprecation. | | **Speed** | Local inference eliminates network round-trip latency. Your model responds in milliseconds, not seconds. | | **Customization** | Fine-tune on your own data for your specific domain. Cloud APIs don't allow this — or charge a premium. | | **Compliance** | Data residency, air-gap, and sovereignty requirements met by design. HIPAA, GDPR, and regulated industries. | | **Cost at scale** | For high-volume use cases, local inference is cheaper than per-token cloud API billing. One hardware investment, unlimited inference. | Vitalik Buterin's [April 2026 post](https://vitalik.eth.limo/general/2026/04/02/secure_llms.html) details the backdoor and censorship risks of hosted models and why local serving matters for sovereignty. ## Pick your runtime | Runtime | Best for | Default port | OpenAI-compatible | | - | - | - | - | | **Ollama** | Easiest start, 1-command model pulls | 11434 | Yes (`/v1`) | | **LM Studio** | GUI-first, beginner-friendly | 1234 | Yes (`/v1`) | | **llama.cpp** | Max control, fine-grained quantization | 8080 | Yes (`/v1`) | | **vLLM** | High-throughput multi-GPU serving | 8000 | Yes (`/v1`) | > [!TIP] > > Start with Ollama. It handles model downloads, quantization, and GPU offloading automatically. Graduate to vLLM when you need concurrent-request throughput. ## Hardware fit Memory is the constraint. Formula: **params × bytes-per-weight + overhead**. | Model size | Q4 memory | Fits on | | - | - | - | | 7–8B | \~5 GB | 8 GB GPU / 16 GB RAM | | 13–14B | \~9 GB | 16 GB GPU / 32 GB RAM | | 32–35B | \~20 GB | 24 GB GPU / 32 GB RAM | | 70B | \~43 GB | 48 GB GPU / 64 GB RAM | | 671B MoE | \~350 GB+ | Multi-GPU server | **Example calc:** 32B params × 0.5 bytes (4-bit) = 16 GB weights + \~4 GB KV cache/runtime = \~20 GB total. See the full [hardware reference](https://docs.tokios.com/guides/hardware-reference.md) and [model fit calculator](https://docs.tokios.com/guides/model-fit.md) for detailed sizing. ## Run your first model **1. Install Ollama** ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} curl -fsSL https://ollama.com/install.sh | sh ``` **2. Pull a model** ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} ollama pull qwen3:8b ``` **3. Run it** ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} ollama run qwen3:8b ``` You now have a private LLM. Your prompts never left your machine. Ollama also serves an OpenAI-compatible API at `http://127.0.0.1:11434/v1` — any OpenAI SDK can call it locally. ## The missing layer: authenticated access Here's the wall every self-hosting guide stops at: your model works on `localhost`, but your **laptop, teammate, coding agent, or n8n workflow** can't reach it securely. * Binding to `0.0.0.0` exposes the model with **no authentication** — anyone on the network can call it. * Port forwarding fails behind CGNAT (Starlink, apartment ISPs). * Tunnels (Cloudflare, ngrok) add buffering, timeouts, and request caps. The Tokios connector adds the missing layer: an **authenticated, HTTPS, OpenAI- and Anthropic-compatible endpoint** — without opening a port. **1. Keep your model on localhost** Default binding. Don't change it. **2. Pair the connector** ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} tokios-connector TKS-XXXX-XXXX ``` Approve in the [dashboard](https://tokios.com/console). **3. Register a deployment** Give your model a public name (e.g., `private-qwen3`). **4. Create scoped API keys** 1 key per teammate or tool. Scope by model-name glob. Revoke instantly. **5. Call from anywhere** ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} curl https://api.tokios.com/v1/chat/completions \ -H "Authorization: Bearer sk-tok-YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "private-qwen3", "messages": [{"role": "user", "content": "Hello!"}]}' ``` You've got a private LLM running. Now give it a secure, authenticated endpoint your team and tools can call from anywhere. ## Migration from cloud APIs If you're currently using OpenAI, Anthropic, or another cloud API, migration is straightforward: 1. **Pick an equivalent open-weight model.** Llama 3.1 70B, Qwen3 72B, or DeepSeek-R1 match GPT-4 on most benchmarks. 2. **Serve it locally** with Ollama or vLLM (see steps above). 3. **Point your existing code** at the local endpoint. Change `base_url` from `https://api.openai.com/v1` to `https://api.tokios.com/v1` (or your local `http://127.0.0.1:PORT/v1`). 4. **Nothing else changes.** The OpenAI SDK, Anthropic SDK, and all compatible clients work identically. Your app code doesn't change. Your prompts don't change. Your workflow doesn't change. The only difference: your data stays on your network. See [Bring your AI in-house](https://docs.tokios.com/guides/bring-your-ai-in-house.md) for the full migration walkthrough. ## FAQ > **What is a private LLM?** > > A language model that runs on hardware you control. Your prompts, outputs, and data never leave your network. Open-weight models (Llama, Qwen, Mistral, DeepSeek) make this possible — you download the weights and serve them yourself. > **Can I self-host ChatGPT?** > > Not ChatGPT itself (it's proprietary), but you can self-host models with comparable quality — Llama 3.1 70B, Qwen3 72B, DeepSeek-R1 70B — and get a ChatGPT-like experience with Open WebUI or any OpenAI-compatible client. > **How much does it cost to self-host an LLM?** > > Hardware: a 16 GB GPU (RTX 4080, \~$1,200) runs 8B models comfortably. A 24 GB RTX 4090 (~$1,600) handles 32B. After hardware, the marginal cost per token is electricity only. No subscription, no per-token billing. > **Is self-hosting an LLM safe?** > > Safer than hosted APIs for data privacy — nothing leaves your network. The risk is exposing the serving port without auth. Keep the model on localhost and use an authenticated endpoint (like Tokios) for remote access. > **Can I run a private LLM offline / air-gapped?** > > Yes. Ollama and llama.cpp work fully offline once the model is downloaded. For air-gapped environments, download the model file on a connected machine and transfer it. The Tokios connector requires outbound internet to reach `api.tokios.com` — for fully air-gapped use, call the model's localhost API directly. > **How do I give my team access to my self-hosted model?** > > Create a separate `sk-tok-…` API key per teammate in the Tokios dashboard. Scope each key to specific deployments. Revoke access instantly without touching the model or server. No shared passwords, no exposed ports. See [Team access to local models](https://docs.tokios.com/use-cases/team-access.md). > **[Team access](https://docs.tokios.com/use-cases/team-access.md)** > > Full walkthrough for giving your team API access to your local models. > **[Bring your AI in-house](https://docs.tokios.com/guides/bring-your-ai-in-house.md)** > > Migration guide for teams switching from cloud APIs to local inference. > **[Ollama remote access](https://docs.tokios.com/guides/ollama-remote-access.md)** > > The full guide to authenticated remote access for Ollama. > **[Quickstart](https://docs.tokios.com/quickstart.md)** > > Pair a connector and register your first model. > **[Model fit guide](https://docs.tokios.com/guides/model-fit.md)** > > Find the right model for your exact hardware. > **[What is a local LLM?](https://docs.tokios.com/guides/what-is-local-llm.md)** > > Complete guide to running large language models locally. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.