Local LLM recipes

Recipe 1: Personal coding assistant (8 GB VRAM, 10 min)

Hardware: 8 GB VRAM GPU (RTX 3060, 4060, or similar) Use case: Code completion and chat inside your editor Backend: Ollama + Continue or Cline

Install Ollama and pull a 7B coder model

# Install Ollama (Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a coding-optimized 7B model
ollama pull qwen2.5-coder:7b

Verify the model responds

curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "messages": [{"role": "user", "content": "Write a Python hello world"}],
    "max_tokens": 100
  }'

Install the Tokios connector and register a deployment

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

In the dashboard, register a deployment named my-coder pointing at upstream qwen2.5-coder:7b. Create an API key.

Configure Continue or Cline

{
  "baseURL": "https://api.tokios.com/v1",
  "apiKey": "sk-tok-YOUR_KEY",
  "model": "my-coder"
}

Expected performance: ~40 tok/s on an RTX 4060. Responsive tab completion and inline edits with 4K context.

Recipe 2: Family chatbot server (16 GB VRAM, 15 min)

Hardware: 16 GB VRAM GPU (RTX 4080, 4070 Ti Super, or similar) Use case: Self-hosted ChatGPT-style interface for household use Backend: Ollama + Open WebUI + Tokios

Start a 14B chat model with Ollama

ollama pull qwen2.5:14b

Run Open WebUI with Docker

docker run -d \
  --name open-webui \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 and create an admin account.

Expose through Tokios for remote access

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Register a deployment named family-chat pointing at qwen2.5:14b. Create an API key for each family member — scoped keys let you revoke access individually.

Expected performance: ~25 tok/s on an RTX 4080 with 8K context. Comfortable for multi-turn conversations.

Create separate API keys per family member so you can revoke one without affecting others.

Recipe 3: Team API endpoint (48 GB+ VRAM, 30 min)

Hardware: 48 GB+ VRAM (A6000, 2x RTX 4090, or A100) Use case: Shared model endpoint for a development team Backend: vLLM + Tokios

Install vLLM and launch a 70B model

pip install vllm

vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --host 127.0.0.1 \
  --port 8000 \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --host 127.0.0.1 \
  --port 8000 \
  --quantization awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

Verify the server responds

curl http://127.0.0.1:8000/v1/models

Install the Tokios connector

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Point the connector at http://127.0.0.1:8000/v1.

Register a deployment named team-70b. In the Keys tab, create 1 key per teammate. Scope each key to the team-* pattern so teammates can only reach shared deployments.

Expected performance: ~30 tok/s aggregate throughput with AWQ quantization. Handles 3–5 concurrent requests with acceptable latency.

With 2 GPUs, tensor parallelism gives better decode throughput than pipeline parallelism on a single node. See Tensor vs pipeline parallelism.

Recipe 4: Document Q&A pipeline (24 GB VRAM, 45 min)

Hardware: 24 GB VRAM GPU (RTX 3090, 4090, or similar) Use case: Ask questions about your PDFs and documents without sending them to the cloud Backend: Ollama (embedding + chat) + custom RAG script + Tokios

Pull an embedding model and a 32B chat model

ollama pull nomic-embed-text
ollama pull qwen2.5:32b

The embedding model (~270 MB) creates vector representations of your documents. The 32B model answers questions grounded in those embeddings.

Build a simple RAG pipeline

import requests, json

EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL  = "qwen2.5:32b"
OLLAMA_URL  = "http://127.0.0.1:11434"

def embed(text: str) -> list[float]:
    r = requests.post(f"{OLLAMA_URL}/api/embeddings",
                       json={"model": EMBED_MODEL, "prompt": text})
    return r.json()["embedding"]

def ask(question: str, context: str) -> str:
    prompt = f"Answer based on this context:\n\n{context}\n\nQuestion: {question}"
    r = requests.post(f"{OLLAMA_URL}/v1/chat/completions",
                       json={"model": CHAT_MODEL,
                             "messages": [{"role": "user", "content": prompt}],
                             "max_tokens": 512})
    return r.json()["choices"][0]["message"]["content"]

Expose the chat model through Tokios

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Register a deployment named doc-qa pointing at qwen2.5:32b. Your RAG script can now call the model through https://api.tokios.com/v1 from any machine, while embeddings stay local.

Expected performance: ~15 tok/s for the 32B model on an RTX 4090. Embedding is near-instant for short passages.

Keep the embedding model local (it is fast and small) and route only the chat model through Tokios. This avoids sending document text over the network for embedding.

Recipe 5: DGX Spark max throughput (128 GB, 20 min)

Hardware: NVIDIA DGX Spark (128 GB unified memory) Use case: Maximum throughput for large MoE models Backend: vLLM + Tokios

Launch a 120B MoE model at FP8

vllm serve Qwen/Qwen3-235B-A22B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --quantization fp8 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 16384 \
  --trust-remote-code

The DGX Spark’s 128 GB unified memory fits models that need 2–3 consumer GPUs. MoE models like Qwen3-235B-A22B only activate ~22B parameters per token, so inference is fast despite the large total parameter count.

Verify and benchmark

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-235B-A22B-FP8",
    "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
    "max_tokens": 256
  }'

Expose through Tokios

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Expected performance: ~35 tok/s on DGX Spark for the active parameter set. 16K context fits comfortably in unified memory.

See vLLM on DGX Spark and DGX Spark memory tuning for detailed configuration options.

Recipe 6: Laptop-friendly offline AI (16 GB RAM, 5 min)

Hardware: Any laptop with 16 GB RAM (no dedicated GPU required) Use case: Private AI assistant that works without internet Backend: Ollama (CPU mode)

Install Ollama and pull a quantized 7B model

# macOS
curl -fsSL https://ollama.com/install.sh | sh

# Or download from https://ollama.com for Windows/Linux
ollama pull qwen2.5:7b-instruct-q4_K_M

Q4_K_M quantization keeps the model under 5 GB, so it runs on CPU with 16 GB RAM.

Chat locally — no internet required

ollama run qwen2.5:7b-instruct-q4_K_M

That is it. The model runs entirely on your laptop. No network calls, no telemetry, no API keys.

Optionally expose through Tokios when online

When you have internet and want to reach the model from another device:

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Register a deployment named laptop-offline. The connector only dials out when the network is available — offline, your model keeps working locally.

Expected performance: ~5 tok/s on CPU (modern laptop). Usable for chat and drafting, not for real-time coding assistance.

If your laptop has an Apple silicon chip, Ollama uses the Neural Engine and Metal automatically — expect 20–30 tok/s on M-series chips.

Recipe 7: Multi-model routing (24 GB+ VRAM, 30 min)

Hardware: 24 GB+ VRAM GPU Use case: Serve 3 models of different sizes through 1 Tokios endpoint Backend: Ollama or vLLM + Tokios with config file

Pull 3 models at different sizes

ollama pull qwen2.5:3b     # fast-small: quick classification
ollama pull qwen2.5:14b    # medium: general chat
ollama pull qwen2.5:32b    # large: complex reasoning

Create a multi-route connector config

Create tokios-connector.json:

{
  "Routes": [
    { "Deployment": "fast-small", "Upstream": "local-ollama" },
    { "Deployment": "medium",     "Upstream": "local-ollama" },
    { "Deployment": "large",      "Upstream": "local-ollama" }
  ],
  "Upstreams": {
    "local-ollama": {
      "BaseUrl": "http://127.0.0.1:11434/v1"
    }
  }
}

In the Models tab, register:

Deployment name	Upstream model id
`fast-small`	`qwen2.5:3b`
`medium`	`qwen2.5:14b`
`large`	`qwen2.5:32b`

Route by sending the model field

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokios.com/v1",
    api_key="sk-tok-YOUR_KEY",
)

# Fast classification
client.chat.completions.create(
    model="fast-small",
    messages=[{"role": "user", "content": "Is this email spam? ..."}],
)

# Complex reasoning
client.chat.completions.create(
    model="large",
    messages=[{"role": "user", "content": "Analyze this contract clause..."}],
)

Expected performance: The 3B model responds in under 1 second. The 32B model takes 2–3 seconds for the first token but produces higher-quality output. Same endpoint, same key, different model field.

Ollama keeps only 1 model in VRAM at a time by default and swaps on request. If you need all 3 loaded simultaneously, run separate backends or use vLLM with LoRA adapters. See Model routing.

Recipe 8: Benchmarking your setup (any hardware, 10 min)

Hardware: Any GPU or CPU running a local model Use case: Compare throughput and latency across models, quantizations, or hardware Backend: Any + Tokios

Start your model server

Use any backend — Ollama, vLLM, llama.cpp, or LM Studio. Confirm it responds on its /v1 endpoint.

Run a throughput benchmark

# Simple throughput test with curl and timing
for i in 1 2 3 4 5; do
  time curl -s http://127.0.0.1:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "qwen2.5:7b",
      "messages": [{"role": "user", "content": "Write a 200-word essay about recursion"}],
      "max_tokens": 256,
      "stream": false
    }' | python3 -c "
import sys, json
r = json.load(sys.stdin)
u = r['usage']
print(f\"Tokens: {u['completion_tokens']}, Time: check wall clock\")
"
done

Benchmark through Tokios to measure end-to-end latency

# Same test through the Tokios endpoint
time curl -s https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-deployment",
    "messages": [{"role": "user", "content": "Write a 200-word essay about recursion"}],
    "max_tokens": 256,
    "stream": false
  }' | python3 -c "
import sys, json
r = json.load(sys.stdin)
u = r['usage']
print(f\"Tokens: {u['completion_tokens']}\")
"

Compare quantizations

Pull the same model at different quantizations and benchmark each:

ollama pull qwen2.5:7b-instruct-q4_K_M
ollama pull qwen2.5:7b-instruct-q5_K_M
ollama pull qwen2.5:7b-instruct-q8_0

Run the same prompt against each and compare tokens per second. Q5_K_M typically gives the best quality-to-speed trade-off at 7B.

Expected results: The tunnel adds 50–150 ms of latency per request (1 round trip to the Tokios gateway). Streaming throughput is nearly identical to local — the tunnel does not buffer SSE.

For a more structured benchmark, see Benchmark your deployment.

Case studies

Real deployment patterns from teams and individuals running local models.

Choose a local model

Pick the right model and quantization for your hardware and workflow.

Model routing

Quickstart

Go from zero to a working connector, deployment, and API key.

Get Started

Models

Concepts

Reference

Local LLM recipes

Next steps

Case studies

Choose a local model

Model routing

Quickstart

​Next steps

Case studies

Choose a local model

Model routing

Quickstart

Next steps