Local LLM case studies

These are composite profiles based on common deployment patterns. Each one captures real trade-offs, numbers, and lessons from running local models in production or personal use.

Solo developer: coding assistant on a MacBook

Setup: Mac Studio M2 Ultra + Ollama + Claude Code via Tokios Model: Qwen3-30B Q5_K_M Time to first working setup: 20 minutes A solo full-stack developer wanted a coding assistant that keeps source code on their machine. They run Ollama on a Mac Studio M2 Ultra with 64 GB unified memory and point Claude Code at it through Tokios.

ollama pull qwen3:30b-a3b-q5_K_M

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model mac-coder

Metric	Value
Tokens per second	~28 tok/s
Monthly cost	$0 (hardware already owned)
Cloud alternative	~$20/month for hosted API
Break-even	Already owned hardware

What worked: The M2 Ultra’s memory bandwidth handles the 30B model well. Tokios means the same setup works from a laptop on the road — just point Claude Code at api.tokios.com. Limitation: Context length tops out around 8K tokens on this model. For large codebases with many files open, the developer chunks work into smaller sessions rather than feeding entire repositories.

Key takeaway: If you already own Apple silicon with 32 GB+ RAM, a local coding assistant costs $0/month and keeps your code private. Tokios adds remote access without a VPN.

Small team: shared model endpoint for 5 developers

Setup: RTX 4090 workstation + vLLM + Tokios connector Model: Qwen3-72B FP8 (AWQ quantization) Time to first working setup: 45 minutes A 5-person backend team shares a single RTX 4090 workstation as a team model server. They use vLLM for throughput and Tokios to give each developer a scoped API key.

vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --host 127.0.0.1 \
  --port 8000 \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 8

{
  "Upstream": {
    "BaseUrl": "http://127.0.0.1:8000/v1"
  }
}

Metric	Value
Concurrent users	5
Aggregate throughput	~30 tok/s
Per-user latency (1 user active)	~800 ms first token
Per-user latency (5 users active)	~2 s first token
VRAM used	~42 GB of 24 GB (overflow to system RAM)

What worked: Consistent model access with no rate limits. Scoped keys mean each developer has their own sk-tok-… key — revoking one doesn’t affect others. Gotcha: The 72B AWQ model barely fits in 24 GB VRAM. During peak concurrent usage, vLLM spills KV cache to CPU memory, which slows throughput. The team added --max-num-seqs 8 to cap concurrent requests and prevent OOM crashes. What they’d change: Move to a 48 GB A6000 or add a second GPU for tensor parallelism. See Tensor vs pipeline parallelism.

Key takeaway: A single consumer GPU can serve a small team, but plan for peak concurrency. Cap --max-num-seqs to prevent OOM and accept higher latency under load.

Homelab enthusiast: DGX Spark as always-on inference server

Setup: DGX Spark + vLLM + Tokios + Tailscale (for LAN access) Models: Rotating collection — Qwen3-30B for daily use, DeepSeek-R1 for reasoning tasks, Phi-4 for quick classification Time to first working setup: 1 hour (including model downloads) A homelab hobbyist runs a DGX Spark 24/7 as a personal inference server. The 128 GB unified memory lets them keep a large model loaded and swap others in as needed. Tokios exposes models to their phone, laptop, and home automation scripts.

vllm serve Qwen/Qwen3-30B-A3B-FP8 \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 16384

{
  "Routes": [
    { "Deployment": "daily",      "Upstream": "vllm-main" },
    { "Deployment": "reasoning",  "Upstream": "vllm-reason" },
    { "Deployment": "classifier", "Upstream": "vllm-fast" }
  ],
  "Upstreams": {
    "vllm-main":   { "BaseUrl": "http://127.0.0.1:8000/v1" },
    "vllm-reason": { "BaseUrl": "http://127.0.0.1:8001/v1" },
    "vllm-fast":   { "BaseUrl": "http://127.0.0.1:8002/v1" }
  }
}

Metric	Value
Idle power draw	~65 W
Under-load power draw	~200 W
Monthly electricity cost	~$8–15 (varies by region)
Uptime	99%+ (systemd auto-restart)

What worked: The DGX Spark’s 128 GB unified memory fits models that would need 2–3 consumer GPUs. Running multiple vLLM instances on different ports lets them serve 3 models simultaneously. Tokios model routing means their phone app just changes the model field to switch between daily chat and deep reasoning. Tip: Use model routing to auto-select a deployment by request type. Point chat clients at daily and automation scripts at classifier.

Key takeaway: A DGX Spark as an always-on server costs $8–15/month in electricity and gives you unlimited inference on models that won’t fit on consumer hardware. The 128 GB unified memory is the real differentiator.

Startup: local models for customer data processing

Setup: 2x A100 80 GB + vLLM + Tokios Models: Llama 3.1 70B for analysis, Phi-4 for classification Time to first working setup: 2 hours (including infrastructure) A 10-person startup processes customer documents that contain sensitive financial data. Compliance requires that data never leaves their infrastructure. They run 2 separate vLLM instances — 1 for analysis, 1 for classification — and expose both through Tokios.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

vllm serve microsoft/phi-4 \
  --host 127.0.0.1 \
  --port 8001 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 4096

{
  "Routes": [
    { "Deployment": "analyzer",   "Upstream": "analysis" },
    { "Deployment": "classifier", "Upstream": "classify" }
  ],
  "Upstreams": {
    "analysis": { "BaseUrl": "http://127.0.0.1:8000/v1" },
    "classify": { "BaseUrl": "http://127.0.0.1:8001/v1" }
  }
}

Metric	Value
Documents processed per day	~2,000
Cost before (cloud GPU)	~$3,000/month
Cost after (owned A100s + Tokios)	~$500/month (amortized hardware + electricity)
Break-even	3 months
Compliance requirement	Data never leaves premises

Architecture: Their processing pipeline sends documents to the analyzer deployment for summarization and entity extraction, then routes extracted fields to the classifier deployment for categorization. Both models run on separate A100s to avoid VRAM contention. What worked: Separate deployments for different tasks means they can scale, restart, or swap models independently. Tokios scoped keys limit the processing service to only the analyzer and classifier deployments. What they’d change: Add a 3rd model for quality checking. The current 2-model pipeline occasionally misclassifies edge cases that a larger reasoning model would catch.

Key takeaway: When compliance requires on-premises data, local models pay for themselves fast. Separate deployments per task give you independent scaling and failure isolation. This startup broke even in 3 months versus cloud GPU costs.

Researcher: benchmarking across quantizations

Setup: RTX 4090 + llama.cpp + lm-evaluation-harness Testing: Same model (Qwen2.5-7B) at Q4_K_M, Q5_K_M, Q8_0, FP16 Time to full benchmark suite: 3 hours An ML researcher needed to measure how quantization affects evaluation scores for a specific task (code generation). They ran the same model at 4 quantization levels and compared perplexity and pass rates.

# From HuggingFace — 4 quantization levels of the same model
huggingface-cli download Qwen/Qwen2.5-7B-Instruct-GGUF \
  qwen2.5-7b-instruct-q4_k_m.gguf \
  qwen2.5-7b-instruct-q5_k_m.gguf \
  qwen2.5-7b-instruct-q8_0.gguf \
  --local-dir ./models

./llama-server \
  -m ./models/qwen2.5-7b-instruct-q5_k_m.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 99 \
  -c 4096

# Point evaluation harness at Tokios endpoint
export OPENAI_API_BASE=https://api.tokios.com/v1
export OPENAI_API_KEY=sk-tok-YOUR_KEY

# Run eval suite
python -m lm_eval --model local-completions \
  --model_args model=bench-q5km,base_url=$OPENAI_API_BASE \
  --tasks humaneval \
  --batch_size 1

Quantization	Model size	Tokens/sec	Eval score	VRAM used
Q4_K_M	4.4 GB	~55	72.1%	5.2 GB
Q5_K_M	5.1 GB	~48	76.8%	6.0 GB
Q8_0	7.7 GB	~38	78.2%	8.5 GB
FP16	14 GB	~25	79.0%	15 GB

Finding: Q5_K_M scored within 2.2 percentage points of FP16 while running almost twice as fast. For this researcher’s code generation tasks, Q5_K_M is the sweet spot. How Tokios helped: Registering each quantization as a separate deployment (bench-q4km, bench-q5km, bench-q8, bench-fp16) let the evaluation harness switch between them by changing only the model field. Results were directly comparable because the harness called the same endpoint shape for each run.

Key takeaway: Q5_K_M is the sweet spot for most 7B evaluation tasks — within 2–3% of FP16 quality at nearly 2x the speed. Register each quantization as a separate Tokios deployment to benchmark them through a consistent API.

Common patterns across all case studies

Pattern	Who uses it	Why
Scoped API keys per user	Teams, startups	Revoke access without affecting others
Separate deployments per task	Teams, researchers	Independent scaling and model swaps
Tokios for remote access	Solo devs, homelab	Reach models from laptop or phone without port forwarding
Quantized models	Everyone	Fit larger models in less VRAM with minimal quality loss
vLLM for throughput	Teams, startups	Better concurrent request handling than Ollama

Get Started

Models

Concepts

Reference

Local LLM case studies

Solo developer: coding assistant on a MacBook

Small team: shared model endpoint for 5 developers

Homelab enthusiast: DGX Spark as always-on inference server

Startup: local models for customer data processing

Researcher: benchmarking across quantizations

Common patterns across all case studies

Next steps

Recipes

Benchmark your deployment

Model routing

Choose a local model

​Solo developer: coding assistant on a MacBook

​Small team: shared model endpoint for 5 developers

​Homelab enthusiast: DGX Spark as always-on inference server

​Startup: local models for customer data processing

​Researcher: benchmarking across quantizations

​Common patterns across all case studies

​Next steps

Recipes

Benchmark your deployment

Model routing

Choose a local model

Solo developer: coding assistant on a MacBook

Small team: shared model endpoint for 5 developers

Homelab enthusiast: DGX Spark as always-on inference server

Startup: local models for customer data processing

Researcher: benchmarking across quantizations

Common patterns across all case studies

Next steps