> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# What is a local LLM (locallm)? The complete guide to running large language models locally

> A local LLM (also written as locallm) runs on your hardware instead of a cloud API. Learn why teams and developers choose local inference for sovereignty, speed, and control — and how to set it up.

A local LLM (also written as **locallm**) is a large language model that runs on your own hardware — your laptop, workstation, server, or homelab GPU box — instead of calling a cloud API like OpenAI or Anthropic. The model weights live on your disk, inference happens on your GPU or CPU, and your data never leaves your network.

This guide covers what local LLMs are, why you'd run one, what hardware you need, and how to get from zero to a working local model with API access in under 30 minutes.

Last verified: 2026-10-24

## What is a local LLM?

A **large language model (LLM)** is a neural network trained on text data that can generate text, answer questions, write code, and perform other language tasks. Popular models include Meta's Llama, Alibaba's Qwen, DeepSeek, Mistral, and Google's Gemma.

A **local LLM** (sometimes written as **locallm** without a space) is simply one of these models running on your own machine instead of being accessed through a cloud provider's API. The key differences:

| | Cloud API (OpenAI, Anthropic) | Local LLM |
| - | - | - |
| **Where it runs** | Provider's data center | Your hardware |
| **Data privacy** | Data sent to third party | Data stays on your network |
| **Latency** | Network round-trip + queue | Local inference only |
| **Rate limits** | Provider-imposed | Limited only by your hardware |
| **Model access** | Provider's model menu | Any open-weight model |
| **Customization** | None (unless fine-tuning API) | Fine-tune on your own data |
| **Cost model** | Per-token pricing | Hardware investment + electricity |

## Why run an LLM locally?

### Sovereignty

Your data never leaves your network. For teams working with sensitive information — medical records, legal documents, proprietary code, financial data — local inference eliminates the risk of data exposure to third-party APIs. There's no prompt logging on someone else's server, no training on your conversations, no compliance audit trail to manage.

### Speed

Local inference eliminates the network round-trip to a cloud API. For latency-sensitive applications — real-time code completion, interactive chat, streaming responses — local models respond in milliseconds instead of the hundreds of milliseconds (or seconds) that cloud APIs can take during peak load.

### Control

No rate limits imposed by a provider. No model deprecation surprises. No policy changes that restrict what you can generate. No API pricing increases. Your model runs when you want it to, how you want it to, for as long as you want.

### Customization

Fine-tune models on your own data for your specific domain. Train a model on your company's codebase, your industry's terminology, your team's communication style. Cloud APIs don't offer this — or they charge a premium for custom fine-tuning endpoints.

### Cost at scale

For high-volume use cases, local inference can be cheaper than paying per-token to a cloud API. The breakeven point depends on your hardware, electricity costs, and usage volume — but for teams sending thousands of requests per day, a one-time hardware investment pays for itself quickly.

## What you need to run a local LLM

### Hardware requirements

The primary constraint is **VRAM** (video RAM on your GPU). Larger models need more VRAM:

| Model size | Minimum VRAM | Example models | Example GPUs |
| - | - | - | - |
| 1B–3B | 4 GB | Qwen2.5-1.5B, Gemma 2-2B | RTX 3060, RTX 4060 |
| 7B–9B | 8 GB | Llama 3.1-8B, Qwen2.5-7B, Gemma 2-9B | RTX 4060 Ti, RTX 3080 |
| 14B | 12–16 GB | Qwen2.5-14B | RTX 4070 Ti Super, RTX 3090 |
| 32B–34B | 24 GB | Llama 3.1-32B, Qwen2.5-32B, CodeLlama-34B | RTX 3090, RTX 4090, RTX A5000 |
| 70B | 48 GB (quantized) or 80 GB (full) | Llama 3.1-70B, Qwen2.5-72B | 2× RTX 3090, RTX A6000, DGX Spark |
| 405B | 320 GB+ | Llama 3.1-405B | Multi-GPU server, Mac Studio M5 Ultra 512 GB |

You can also run models on **CPU only** (without a GPU), but inference will be significantly slower — usable for testing and light workloads, not for production or interactive use.

For detailed hardware guidance, see [Hardware reference](https://docs.tokios.com/guides/hardware-reference.md) and [Model fit calculator](https://docs.tokios.com/guides/model-fit.md).

### Model selection

Open-weight models are available in a range of sizes and specializations:

* **General purpose:** Llama 3.1 (8B, 70B, 405B), Qwen2.5 (7B, 14B, 32B, 72B), Mistral 7B
* **Code:** Qwen2.5-Coder (7B, 32B), CodeLlama (7B, 13B, 34B), DeepSeek-Coder
* **Reasoning:** DeepSeek-R1, QwQ-32B
* **Vision:** Qwen2.5-VL, Llama 3.2 Vision
* **Small and fast:** Gemma 2 (2B, 9B), Phi-3 (3.8B, 14B), Qwen2.5-1.5B

For help choosing, see [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md) and [Models by task](https://docs.tokios.com/guides/model-by-task.md).

### Inference engine

You need software to load the model and serve it. The major options:

| Engine | Strengths | Best for |
| - | - | - |
| **Ollama** | Easiest setup, broad model support | Personal use, quick prototyping |
| **vLLM** | High throughput, production-grade | Teams, production workloads |
| **llama.cpp** | CPU-friendly, quantization support | Low-VRAM hardware, edge devices |
| **LM Studio** | Desktop GUI, model browser | Non-technical users, exploration |
| **SGLang** | High performance, structured generation | Research, complex workflows |
| **Xinference** | Multi-model serving, distributed | Multi-model deployments |

For detailed comparisons, see [Supported backends](https://docs.tokios.com/setup/supported-backends.md) and the per-runtime guides in the [Local LLM Playbook](https://docs.tokios.com/index.md).

## Getting started: from zero to local LLM in 30 minutes

The fastest path is Ollama on a machine with a GPU.

**1. Install Ollama**

```bash Linux theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://ollama.com/install.sh | sh
```

```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}}
brew install ollama
```

Windows: download from [ollama.com](https://ollama.com/download).

**2. Pull a model**

Start with a 7B model if you have 8 GB VRAM, or a 14B model if you have 16 GB:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama pull qwen2.5:7b
```

or

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
ollama pull qwen2.5:14b
```

**3. Verify the model responds**

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:7b",
    "messages": [{"role": "user", "content": "Hello, what can you do?"}],
    "max_tokens": 100
  }'
```

You should get a JSON response with the model's output.

**4. Expose it through an API (optional, for team access)**

If you want your team or external apps to access the model, install the Tokios connector:

```bash Linux theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

Get your claim code `TKS-XXXX-XXXX` from the [Tokios dashboard](https://tokios.com/console). The connector creates an outbound tunnel so your team can call `https://api.tokios.com/v1` with an API key — no port forwarding, no Tailscale, no firewall changes.

See [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) for the full walkthrough.

## Common use cases

### Personal AI assistant

Run a model on your laptop or desktop for private, unlimited use. No API costs, no rate limits, no data leaving your machine. Use it for writing, brainstorming, coding, research, or any task where you'd use ChatGPT but want full privacy.

### Team access to shared models

Run a powerful model on a shared server (homelab GPU box, workstation, DGX Spark) and give your team API access. Each teammate gets their own API key scoped to specific models. See [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) for the full setup.

### Development and testing

Test your application against a local model before deploying to production. Use the same OpenAI-compatible API interface, but with a local backend that you control. No API costs during development, no rate limits during testing.

### Fine-tuning and custom models

Fine-tune a model on your own data for your specific domain. Train a model on your company's codebase, your industry's terminology, your team's communication style. Deploy the fine-tuned model locally and access it through the same API.

### Air-gapped and regulated environments

Run models in environments with no internet access or strict data residency requirements. Government, healthcare, finance, and defense organizations often need local inference because cloud APIs are not permitted.

### Edge and IoT deployments

Run smaller models on edge devices — Raspberry Pi, Jetson, industrial controllers — for local inference without cloud connectivity. Useful for manufacturing, agriculture, field research, and other remote scenarios.

## Challenges and solutions

### Hardware limitations

**Problem:** You don't have enough VRAM for the model you want.

**Solutions:**

* Use a smaller model (7B instead of 70B)
* Quantize the model (4-bit instead of full precision) — see [Quantization guide](https://docs.tokios.com/models/quantization-and-hardware.md)
* Use CPU inference with llama.cpp (slower but works)
* Split the model across multiple GPUs — see [Model sharding](https://docs.tokios.com/guides/model-sharding.md)

### Networking complexity

**Problem:** Your team needs remote access, but you can't open inbound ports or configure Tailscale for everyone.

**Solution:** Use Tokios — the connector dials out through an outbound-only tunnel, so no inbound ports are needed. Works behind CGNAT, corporate firewalls, and NAT. See [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) and [Serve a local LLM from behind CGNAT](https://docs.tokios.com/guides/llm-behind-cgnat.md).

### API compatibility

**Problem:** Your application uses the OpenAI SDK, but you want to call a local model.

**Solution:** Most local inference engines (Ollama, vLLM, llama.cpp, LM Studio) expose an OpenAI-compatible `/v1` API. Point your client at `http://127.0.0.1:PORT/v1` and it works. If you're using Tokios, the endpoint is `https://api.tokios.com/v1` — same interface, remote access.

### Team access and authentication

**Problem:** Your local model has no authentication. Anyone on the network can call it.

**Solution:** Use Tokios to add per-user API keys with model scoping. Each teammate gets their own key, and you can revoke access for one person without disrupting everyone else. See [Add API keys to local models](https://docs.tokios.com/guides/add-api-keys-to-local-models.md).

### Model selection paralysis

**Problem:** There are hundreds of models. Which one should you use?

**Solution:** Start with Qwen2.5-7B (general purpose, 8 GB VRAM) or Qwen2.5-Coder-7B (code tasks, 8 GB VRAM). Upgrade to larger models as you understand your requirements. See [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md) and [Models by task](https://docs.tokios.com/guides/model-by-task.md).

## Next steps

* [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md) — how to pick the right model for your use case
* [Hardware reference](https://docs.tokios.com/guides/hardware-reference.md) — detailed hardware requirements and recommendations
* [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) — give your team API access to your local model
* [Local LLM recipes](https://docs.tokios.com/guides/local-llm-recipes.md) — copy-paste configurations for common setups
* [Supported backends](https://docs.tokios.com/setup/supported-backends.md) — Ollama, vLLM, llama.cpp, LM Studio, and more

## FAQ

> **Is a local LLM as good as ChatGPT?**
>
> For many tasks, yes. Open-weight models like Llama 3.1-70B, Qwen2.5-72B, and DeepSeek-R1 match or exceed GPT-4 on many benchmarks. For general chat, code generation, and reasoning, a well-chosen local model delivers comparable quality. The gap is narrowing rapidly — see [What are evals?](https://docs.tokios.com/guides/what-are-evals.md) for how models are compared.

> **Can I use a local LLM without a GPU?**
>
> Yes, but it will be slow. CPU inference works with llama.cpp and Ollama, but expect 5–10× slower performance compared to GPU inference. Usable for testing and light workloads, not for production or interactive use. A GPU with 8 GB VRAM is the practical minimum for a good experience.

> **How much does it cost to run a local LLM?**
>
> The model itself is free (open-weight). Your costs are hardware (GPU, RAM, storage) and electricity. A workstation with an RTX 4090 costs \~$2,000–$3,000. Electricity for 24/7 inference is \~$50–$100/month depending on your local rate. For high-volume use cases, this is cheaper than paying per-token to a cloud API.

> **Can I fine-tune a local LLM on my own data?**
>
> Yes. Tools like Unsloth, Axolotl, and LLaMA-Factory let you fine-tune open-weight models on your own data. You need a GPU with enough VRAM to load the model plus training data (typically 24 GB for 7B models, 48+ GB for 13B models). See [Serve your fine-tune](https://docs.tokios.com/guides/serve-your-fine-tune.md) for deployment guidance.

> **What's the difference between a local LLM and an API like OpenAI?**
>
> A local LLM runs on your hardware — you control the model, the data, and the infrastructure. An API like OpenAI runs on their servers — you send data over the internet, they run inference, and you pay per token. Local gives you sovereignty, speed, and control. Cloud gives you convenience and unlimited scale (at a cost).

> **Can my team access my local LLM remotely?**
>
> Yes. Use Tokios to create an outbound tunnel from your local model to `https://api.tokios.com/v1`. Your team calls that endpoint with their API key — no port forwarding, no Tailscale, no firewall changes. See [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) for the full setup.

> **What is locallm? Is it different from local LLM?**
>
> No difference — "locallm" is just "local LLM" written as a single word. You'll see both spellings in searches, community discussions (like r/LocalLLaMA), and project names. Some tools and libraries use "locallm" as a package name (e.g., the Python `locallm` package), but they all refer to the same concept: running a large language model on your own hardware instead of a cloud API.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.