> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Bring your AI in-house: migrate from cloud APIs to local inference

> Migration guide for teams using OpenAI, Anthropic, or cloud APIs who want sovereignty, control, and custom models on their own hardware. Same API experience, zero data leaving your network.

You're using OpenAI, Anthropic, or another cloud API. It works. But your data leaves your network on every request. You don't control the model. You can't fine-tune on your own data. Rate limits, policy changes, and model deprecation are someone else's decision.

This guide walks you through migrating from cloud APIs to local inference (running a locallm on your own hardware) — keeping the same developer experience, the same API interface, and the same client code. The only difference: your prompts, your outputs, and your data stay on your hardware.

Last verified: 2026-10-24

## Why bring AI in-house?

### Sovereignty

Your data never leaves your network. No third-party logging. No training on your prompts. No compliance audit trail to manage. For teams working with sensitive information — medical records, legal documents, proprietary code, financial data — local inference eliminates the risk of data exposure.

### Control

No rate limits imposed by a provider. No model deprecation surprises. No policy changes that restrict what you can generate. No API pricing increases. Your model runs when you want it to, how you want it to, for as long as you want.

### Customization

Fine-tune models on your own data for your specific domain. Train a model on your company's codebase, your industry's terminology, your team's communication style. Cloud APIs don't offer this — or they charge a premium for custom fine-tuning endpoints.

### Performance parity

Open-weight models like Llama 3.1 70B, Qwen3 72B, and DeepSeek-R1 match or exceed GPT-4 on many benchmarks. For general chat, code generation, and reasoning, a well-chosen local model delivers comparable quality. The gap is narrowing rapidly — see [What are evals?](https://docs.tokios.com/guides/what-are-evals.md) for how models are compared.

### Speed

Local inference eliminates the network round-trip to a cloud API. For latency-sensitive applications — real-time code completion, interactive chat, streaming responses — local models respond in milliseconds instead of the hundreds of milliseconds (or seconds) that cloud APIs can take during peak load.

## What stays the same

The migration is easier than you think because **nothing in your application code changes**:

* **Same API interface.** OpenAI-compatible `/v1` endpoint. Your existing OpenAI SDK, Anthropic SDK, or HTTP client works without modification.
* **Same request format.** JSON payloads with `model`, `messages`, `max_tokens` — identical to what you're sending to OpenAI today.
* **Same response format.** JSON responses with `choices`, `usage`, `id` — parse them the same way.
* **Same client libraries.** Python SDK, JavaScript SDK, curl, Continue, Cline, Claude Code, Codex — all work.

The only change: you point your client at a different base URL.

## Migration walkthrough

**1. Choose your model**

Pick an open-weight model that matches your current cloud API usage:

| Cloud API | Local equivalent | Hardware needed |
| - | - | - |
| GPT-4o / GPT-4 | Llama 3.1 70B, Qwen3 72B | 48 GB VRAM (RTX A6000, 2× RTX 3090) |
| GPT-4o-mini / GPT-3.5 | Qwen3 14B, Llama 3.1 8B | 16 GB VRAM (RTX 4080, RTX 4070 Ti) |
| Claude 3.5 Sonnet | Qwen3 32B, Llama 3.1 32B | 24 GB VRAM (RTX 4090, RTX 3090) |
| Claude 3 Haiku | Qwen3 7B, Gemma 2 9B | 8 GB VRAM (RTX 4060, RTX 3060) |

See [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md) and [Models by task](https://docs.tokios.com/guides/model-by-task.md) for detailed guidance.

**2. Set up the runtime**

Install Ollama (easiest) or vLLM (highest throughput):

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull your model
ollama pull qwen3:32b
```

Verify it responds:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:32b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

**3. Update your client code**

Change the `base_url` in your OpenAI SDK client:

```python theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Before (cloud API)
from openai import OpenAI
client = OpenAI(api_key="sk-...")  # OpenAI API key

# After (local inference)
from openai import OpenAI
client = OpenAI(
    base_url="http://127.0.0.1:11434/v1",  # Local Ollama
    api_key="not-needed",  # Ollama doesn't require auth
)
```

For team access with authentication, use Tokios:

```python theme={"theme":{"light":"github-light","dark":"css-variables"}}
# With Tokios (authenticated, remote access)
from openai import OpenAI
client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)
```

**That's it.** Nothing else in your code changes. The `model` field, the `messages` array, the response parsing — all identical.

**4. Set up team access (optional)**

If your team needs remote access to the local model, install the Tokios connector:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX
```

Get your claim code from the [dashboard](https://tokios.com/console). The connector creates an outbound tunnel so your team can call `https://api.tokios.com/v1` with per-user API keys — no port forwarding, no Tailscale, no firewall changes.

See [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) for the full walkthrough.

**5. Test and validate**

Run your existing test suite against the local model. Compare outputs to your cloud API baseline. For most tasks (chat, code generation, summarization, extraction), the quality is comparable. For specialized tasks, you may need to fine-tune — see [Serve your fine-tune](https://docs.tokios.com/guides/serve-your-fine-tune.md).

## Client integration examples

```python OpenAI SDK theme={"theme":{"light":"github-light","dark":"css-variables"}}
from openai import OpenAI

# Point at your local model
client = OpenAI(
    base_url="https://api.tokios.com/v1",  # or http://127.0.0.1:11434/v1
    api_key="sk-tok-YOUR_KEY",  # or "not-needed" for local Ollama
)

response = client.chat.completions.create(
    model="qwen3-32b",  # your deployment name
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)
```

```javascript OpenAI SDK theme={"theme":{"light":"github-light","dark":"css-variables"}}
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://api.tokios.com/v1',
  apiKey: 'sk-tok-YOUR_KEY',
});

const response = await client.chat.completions.create({
  model: 'qwen3-32b',
  messages: [{ role: 'user', content: 'Hello' }],
});
console.log(response.choices[0].message.content);
```

```bash curl theme={"theme":{"light":"github-light","dark":"css-variables"}}
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-32b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

## Hardware planning

Your hardware determines what models you can run. The primary constraint is **VRAM** (video RAM on your GPU):

| Model size | Minimum VRAM | Example hardware | Cost range |
| - | - | - | - |
| 7B–8B | 8 GB | RTX 4060, RTX 3060 | $300–$500 |
| 14B | 16 GB | RTX 4080, RTX 4070 Ti | $600–$1,200 |
| 32B | 24 GB | RTX 4090, RTX 3090 | $1,600–$2,000 |
| 70B | 48 GB | RTX A6000, 2× RTX 3090 | $4,000–$8,000 |
| 70B+ | 80+ GB | DGX Spark, Mac Studio M5 Ultra | $3,000–$8,500 |

For detailed hardware guidance, see [Hardware reference](https://docs.tokios.com/guides/hardware-reference.md) and [Model fit calculator](https://docs.tokios.com/guides/model-fit.md).

> [!TIP]
>
> Start with a 24 GB RTX 4090 or used RTX 3090 (\~$1,600–$2,000). It runs 32B models comfortably and 70B models at Q4 quantization. This is the sweet spot for most teams migrating from cloud APIs.

## Cost comparison

For high-volume use cases, local inference is cheaper than cloud APIs. The breakeven point depends on your usage:

| Usage level | Cloud API cost (monthly) | Local inference cost (monthly) |
| - | - | - |
| Light (100K tokens/day) | \~$30–$60 | \~$5–$10 (electricity) |
| Medium (1M tokens/day) | \~$300–$600 | \~$30–$50 (electricity) |
| Heavy (10M tokens/day) | \~$3,000–$6,000 | \~$200–$400 (electricity) |

Hardware cost amortized over 3 years:

* RTX 4090 ($2,000) = ~$56/month
* DGX Spark ($3,000) = ~$83/month
* Mac Studio M5 Ultra ($8,500) = ~$236/month

For teams sending millions of tokens per day, the hardware investment pays for itself in 3-6 months. After that, your marginal cost per token is electricity only.

## Common concerns

### "What if the local model isn't as good as GPT-4?"

For most tasks, open-weight models match or exceed GPT-4. Llama 3.1 70B, Qwen3 72B, and DeepSeek-R1 are competitive on reasoning, code generation, and chat. For specialized tasks, you can fine-tune on your own data — something cloud APIs don't allow (or charge a premium for).

See [What are evals?](https://docs.tokios.com/guides/what-are-evals.md) for benchmark comparisons.

### "What about uptime and reliability?"

Your local model is as reliable as your hardware. If your GPU server is always on, your model is always available. No "server busy" errors, no rate limits, no downtime from cloud provider outages. For production workloads, run your model on a dedicated server with UPS backup.

### "What if I need to scale?"

Scale by adding hardware. Need more throughput? Add more GPUs or upgrade to a more powerful GPU. Need to serve more users? Use Tokios to create per-user API keys and route requests to multiple model instances. See [Model routing](https://docs.tokios.com/guides/model-routing.md) and [Team access to local models](https://docs.tokios.com/use-cases/team-access.md).

### "What about fine-tuning?"

Fine-tune on your own data using tools like Unsloth, Axolotl, or LLaMA-Factory. You need a GPU with enough VRAM to load the model plus training data (typically 24 GB for 7B models, 48+ GB for 13B models). Serve the fine-tuned model through the same API. See [Serve your fine-tune](https://docs.tokios.com/guides/serve-your-fine-tune.md).

## Next steps

* [Self-hosted LLM guide](https://docs.tokios.com/guides/self-hosted-private-llm.md) — complete walkthrough from zero to private AI
* [Team access to local models](https://docs.tokios.com/use-cases/team-access.md) — give your team API access
* [Local LLM API](https://docs.tokios.com/guides/local-llm-api.md) — expose your models with an OpenAI-compatible endpoint
* [Choose a local model](https://docs.tokios.com/guides/choose-a-local-model.md) — pick the right model for your use case
* [Hardware reference](https://docs.tokios.com/guides/hardware-reference.md) — detailed hardware requirements

## FAQ

> **Do I need to rewrite my application code?**
>
> No. Your existing OpenAI SDK, Anthropic SDK, or HTTP client works without modification. The only change is the `base_url` — point it at your local model or Tokios endpoint instead of the cloud API.

> **What if I need features specific to OpenAI or Anthropic?**
>
> Most features (chat, code generation, function calling, structured output) are supported by open-weight models. For specialized features (DALL-E image generation, Whisper audio transcription), you may need to use separate local models or keep those specific calls on the cloud API.

> **Can I use both local and cloud APIs?**
>
> Yes. Use local inference for high-volume, privacy-sensitive tasks. Use cloud APIs for specialized features or when you need access to the latest models before they're available as open weights. Tokios can route to local models; your cloud API calls go directly to the provider.

> **What about compliance and regulation?**
>
> Local inference meets data residency, air-gap, and sovereignty requirements by design. Your data never leaves your network. For HIPAA, GDPR, and regulated industries, this is often the only acceptable architecture. See [Self-hosted LLM guide](https://docs.tokios.com/guides/self-hosted-private-llm.md) for compliance details.

> **How do I handle authentication for my team?**
>
> Use Tokios to create per-user API keys with model scoping. Each teammate gets their own key, and you can revoke access for one person without disrupting everyone else. See [Team access to local models](https://docs.tokios.com/use-cases/team-access.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.