Bring your AI in-house: migrate from cloud APIs to local inference

You’re using OpenAI, Anthropic, or another cloud API. It works. But your data leaves your network on every request. You don’t control the model. You can’t fine-tune on your own data. Rate limits, policy changes, and model deprecation are someone else’s decision. This guide walks you through migrating from cloud APIs to local inference (running a locallm on your own hardware) — keeping the same developer experience, the same API interface, and the same client code. The only difference: your prompts, your outputs, and your data stay on your hardware. Last verified: 2026-10-24

Why bring AI in-house?

Sovereignty

Your data never leaves your network. No third-party logging. No training on your prompts. No compliance audit trail to manage. For teams working with sensitive information — medical records, legal documents, proprietary code, financial data — local inference eliminates the risk of data exposure.

Control

No rate limits imposed by a provider. No model deprecation surprises. No policy changes that restrict what you can generate. No API pricing increases. Your model runs when you want it to, how you want it to, for as long as you want.

Customization

Fine-tune models on your own data for your specific domain. Train a model on your company’s codebase, your industry’s terminology, your team’s communication style. Cloud APIs don’t offer this — or they charge a premium for custom fine-tuning endpoints.

Performance parity

Open-weight models like Llama 3.1 70B, Qwen3 72B, and DeepSeek-R1 match or exceed GPT-4 on many benchmarks. For general chat, code generation, and reasoning, a well-chosen local model delivers comparable quality. The gap is narrowing rapidly — see What are evals? for how models are compared.

Speed

Local inference eliminates the network round-trip to a cloud API. For latency-sensitive applications — real-time code completion, interactive chat, streaming responses — local models respond in milliseconds instead of the hundreds of milliseconds (or seconds) that cloud APIs can take during peak load.

What stays the same

The migration is easier than you think because nothing in your application code changes:

Same API interface. OpenAI-compatible /v1 endpoint. Your existing OpenAI SDK, Anthropic SDK, or HTTP client works without modification.
Same request format. JSON payloads with model, messages, max_tokens — identical to what you’re sending to OpenAI today.
Same response format. JSON responses with choices, usage, id — parse them the same way.
Same client libraries. Python SDK, JavaScript SDK, curl, Continue, Cline, Claude Code, Codex — all work.

The only change: you point your client at a different base URL.

Migration walkthrough

Choose your model

Pick an open-weight model that matches your current cloud API usage:

Cloud API	Local equivalent	Hardware needed
GPT-4o / GPT-4	Llama 3.1 70B, Qwen3 72B	48 GB VRAM (RTX A6000, 2× RTX 3090)
GPT-4o-mini / GPT-3.5	Qwen3 14B, Llama 3.1 8B	16 GB VRAM (RTX 4080, RTX 4070 Ti)
Claude 3.5 Sonnet	Qwen3 32B, Llama 3.1 32B	24 GB VRAM (RTX 4090, RTX 3090)
Claude 3 Haiku	Qwen3 7B, Gemma 2 9B	8 GB VRAM (RTX 4060, RTX 3060)

See Choose a local model and Models by task for detailed guidance.

Set up the runtime

Install Ollama (easiest) or vLLM (highest throughput):

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull your model
ollama pull qwen3:32b

Verify it responds:

curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:32b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Update your client code

Change the base_url in your OpenAI SDK client:

# Before (cloud API)
from openai import OpenAI
client = OpenAI(api_key="sk-...")  # OpenAI API key

# After (local inference)
from openai import OpenAI
client = OpenAI(
    base_url="http://127.0.0.1:11434/v1",  # Local Ollama
    api_key="not-needed",  # Ollama doesn't require auth
)

For team access with authentication, use Tokios:

# With Tokios (authenticated, remote access)
from openai import OpenAI
client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

That’s it. Nothing else in your code changes. The model field, the messages array, the response parsing — all identical.

Set up team access (optional)

If your team needs remote access to the local model, install the Tokios connector:

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Get your claim code from the dashboard. The connector creates an outbound tunnel so your team can call https://api.tokios.com/v1 with per-user API keys — no port forwarding, no Tailscale, no firewall changes.See Team access to local models for the full walkthrough.

Test and validate

Run your existing test suite against the local model. Compare outputs to your cloud API baseline. For most tasks (chat, code generation, summarization, extraction), the quality is comparable. For specialized tasks, you may need to fine-tune — see Serve your fine-tune.

Client integration examples

from openai import OpenAI

# Point at your local model
client = OpenAI(
    base_url="https://api.tokios.com/v1",  # or http://127.0.0.1:11434/v1
    api_key="sk-tok-YOUR_KEY",  # or "not-needed" for local Ollama
)

response = client.chat.completions.create(
    model="qwen3-32b",  # your deployment name
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://api.tokios.com/v1',
  apiKey: 'sk-tok-YOUR_KEY',
});

const response = await client.chat.completions.create({
  model: 'qwen3-32b',
  messages: [{ role: 'user', content: 'Hello' }],
});
console.log(response.choices[0].message.content);

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-32b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Hardware planning

Your hardware determines what models you can run. The primary constraint is VRAM (video RAM on your GPU):

Model size	Minimum VRAM	Example hardware	Cost range
7B–8B	8 GB	RTX 4060, RTX 3060	$300–$500
14B	16 GB	RTX 4080, RTX 4070 Ti	$600–$1,200
32B	24 GB	RTX 4090, RTX 3090	$1,600–$2,000
70B	48 GB	RTX A6000, 2× RTX 3090	$4,000–$8,000
70B+	80+ GB	DGX Spark, Mac Studio M5 Ultra	$3,000–$8,500

For detailed hardware guidance, see Hardware reference and Model fit calculator.

Start with a 24 GB RTX 4090 or used RTX 3090 (~$1,600–$2,000). It runs 32B models comfortably and 70B models at Q4 quantization. This is the sweet spot for most teams migrating from cloud APIs.

Cost comparison

For high-volume use cases, local inference is cheaper than cloud APIs. The breakeven point depends on your usage:

Usage level	Cloud API cost (monthly)	Local inference cost (monthly)
Light (100K tokens/day)	~$30–$60	~$5–$10 (electricity)
Medium (1M tokens/day)	~$300–$600	~$30–$50 (electricity)
Heavy (10M tokens/day)	~$3,000–$6,000	~$200–$400 (electricity)

Hardware cost amortized over 3 years:

RTX 4090 ($2,000) = ~$56/month
DGX Spark ($3,000) = ~$83/month
Mac Studio M5 Ultra ($8,500) = ~$236/month

For teams sending millions of tokens per day, the hardware investment pays for itself in 3-6 months. After that, your marginal cost per token is electricity only.

Common concerns

”What if the local model isn’t as good as GPT-4?”

For most tasks, open-weight models match or exceed GPT-4. Llama 3.1 70B, Qwen3 72B, and DeepSeek-R1 are competitive on reasoning, code generation, and chat. For specialized tasks, you can fine-tune on your own data — something cloud APIs don’t allow (or charge a premium for). See What are evals? for benchmark comparisons.

”What about uptime and reliability?”

Your local model is as reliable as your hardware. If your GPU server is always on, your model is always available. No “server busy” errors, no rate limits, no downtime from cloud provider outages. For production workloads, run your model on a dedicated server with UPS backup.

”What if I need to scale?”

Scale by adding hardware. Need more throughput? Add more GPUs or upgrade to a more powerful GPU. Need to serve more users? Use Tokios to create per-user API keys and route requests to multiple model instances. See Model routing and Team access to local models.

”What about fine-tuning?”

Fine-tune on your own data using tools like Unsloth, Axolotl, or LLaMA-Factory. You need a GPU with enough VRAM to load the model plus training data (typically 24 GB for 7B models, 48+ GB for 13B models). Serve the fine-tuned model through the same API. See Serve your fine-tune.

Next steps

Self-hosted LLM guide — complete walkthrough from zero to private AI
Team access to local models — give your team API access
Local LLM API — expose your models with an OpenAI-compatible endpoint
Choose a local model — pick the right model for your use case
Hardware reference — detailed hardware requirements

FAQ

Do I need to rewrite my application code?

No. Your existing OpenAI SDK, Anthropic SDK, or HTTP client works without modification. The only change is the base_url — point it at your local model or Tokios endpoint instead of the cloud API.

What if I need features specific to OpenAI or Anthropic?

Most features (chat, code generation, function calling, structured output) are supported by open-weight models. For specialized features (DALL-E image generation, Whisper audio transcription), you may need to use separate local models or keep those specific calls on the cloud API.

Can I use both local and cloud APIs?

Yes. Use local inference for high-volume, privacy-sensitive tasks. Use cloud APIs for specialized features or when you need access to the latest models before they’re available as open weights. Tokios can route to local models; your cloud API calls go directly to the provider.

What about compliance and regulation?

Local inference meets data residency, air-gap, and sovereignty requirements by design. Your data never leaves your network. For HIPAA, GDPR, and regulated industries, this is often the only acceptable architecture. See Self-hosted LLM guide for compliance details.

How do I handle authentication for my team?

Use Tokios to create per-user API keys with model scoping. Each teammate gets their own key, and you can revoke access for one person without disrupting everyone else. See Team access to local models.

Getting started with local LLMs

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Bring your AI in-house: migrate from cloud APIs to local inference

Why bring AI in-house?

Sovereignty

Control

Customization

Performance parity

Speed

What stays the same

Migration walkthrough

Client integration examples

Hardware planning

Cost comparison

Common concerns

”What if the local model isn’t as good as GPT-4?”

”What about uptime and reliability?”

”What if I need to scale?”

”What about fine-tuning?”

Next steps

FAQ

​Why bring AI in-house?

​Sovereignty

​Control

​Customization

​Performance parity

​Speed

​What stays the same

​Migration walkthrough

​Client integration examples

​Hardware planning

​Cost comparison

​Common concerns

​”What if the local model isn’t as good as GPT-4?”

​”What about uptime and reliability?”

​”What if I need to scale?”

​”What about fine-tuning?”

​Next steps

​FAQ

Why bring AI in-house?

Sovereignty

Control

Customization

Performance parity

Speed

What stays the same

Migration walkthrough

Client integration examples

Hardware planning

Cost comparison

Common concerns

”What if the local model isn’t as good as GPT-4?”

”What about uptime and reliability?”

”What if I need to scale?”

”What about fine-tuning?”

Next steps

FAQ