Skip to main content

​
What a DIY stack looks like

A production-ready local LLM API has 7 layers. Here’s what each one does and what people typically use: Each layer is a separate tool to install, configure, update, and debug. They interact in non-obvious ways — nginx buffering affects SSE streaming, tunnel timeouts interact with proxy timeouts, and auth middleware needs to understand the API protocol.

​
Setup comparison

​
Where DIY wins

  • Full control. You own every layer. You can inspect, modify, and debug anything. No opaque managed service between your client and your model.
  • No third-party data path. All traffic stays on infrastructure you control. For regulated environments (healthcare, finance, government), this may be a hard requirement — not a preference.
  • No vendor lock-in. If Tokios shuts down or changes pricing, you’re stuck. A DIY stack you built, you can maintain indefinitely.
  • Unlimited customization. Custom rate limiting, your own logging pipeline, model-specific routing rules, A/B testing — anything is possible when you control the proxy config.
  • Potentially lower cost at scale. For 100+ users, the per-seat cost of a managed service can exceed the cost of self-hosted infrastructure. At scale, DIY economics improve.

​
Where Tokios wins

  • Minutes vs hours. The entire Tokios setup — install connector, register model, create key — takes 2–5 minutes. A DIY stack takes 4–8 hours for a first-time setup, even for someone who knows the tools.
  • No maintenance burden. Certificates renew, tunnels reconnect, and the gateway stays updated without your involvement. A DIY stack needs ongoing attention: cert expirations, tunnel crashes, security patches on nginx.
  • Built-in streaming. SSE streaming works correctly out of the box. In a DIY stack, you need to explicitly disable proxy_buffering in nginx, configure proxy_read_timeout, and verify that your tunnel doesn’t buffer responses.
  • Built-in auth. Per-user API keys with deployment scoping. In DIY, you write or deploy an auth middleware, handle key storage securely, and build revocation logic.
  • CGNAT handling. Tokios’s outbound-only connection works behind any NAT. A DIY tunnel setup may or may not handle CGNAT depending on the tool — Cloudflare Tunnel does, but nginx + Let’s Encrypt alone does not.
  • No security surface to manage. No exposed ports, no certificate lifecycle, no auth middleware vulnerabilities. The connector dials out; the gateway handles everything else.

​
The streaming pitfalls

This is where DIY stacks most commonly break for LLM workloads. 3 separate layers can buffer or time out a streaming response:
  1. nginx buffering. By default, nginx buffers upstream responses. You need proxy_buffering off; and proxy_cache off; in the location block for your LLM endpoint. Miss this, and tokens dump in one burst at the end.
  2. Proxy timeouts. nginx’s default proxy_read_timeout is 60 seconds. A long generation on consumer hardware can exceed that. You need to set it to 300s or more. If a tunnel sits in front (Cloudflare Tunnel), its ~100–120s timeout becomes the new ceiling.
  3. SSE configuration. The Content-Type: text/event-stream header must propagate from your backend through every proxy layer to the client. If any layer strips or overrides it, downstream buffering may kick in.
Each of these is solvable — but they’re 3 separate configs across 2–3 tools, and the failure mode is silent: streaming appears to work but actually delivers all tokens at once, or cuts off mid-generation with a 524 or 504 error. Tokios’s gateway handles all 3 correctly by default because it’s designed for this specific traffic pattern.

​
When DIY makes sense

  • Regulated environments. Healthcare, finance, or government work where no third-party data path is permitted. All traffic must stay on infrastructure you control and can audit.
  • Existing DevOps capacity. You already run Kubernetes, have monitoring in place, and adding one more service is incremental work — not a new skill set.
  • Need full control. Custom routing logic, a logging pipeline beyond Tokios’s built-in opt-in request logging, or integration with internal systems that a managed service can’t accommodate.
  • 100+ users where per-seat cost matters. At high user counts, self-hosted infrastructure can be significantly cheaper than per-seat managed pricing.

​
When Tokios makes sense

  • Want the outcome without building the stack. Your goal is “my model is reachable with an API key” — not “I want to learn nginx config.”
  • Small teams. 2–10 people who need model access. It’s free on the community plan, and nobody’s time goes to infrastructure maintenance.
  • Behind CGNAT. Your ISP blocks inbound connections. A DIY tunnel can work, but Tokios handles it natively with no extra configuration.
  • Need fast setup. You want to be making API calls in 5 minutes, not spending an afternoon configuring proxy layers.

Tokios vs LiteLLM

When an open-source proxy/gateway is the right choice vs a managed service.

When to use Tokios

An honest guide to when Tokios is — and isn’t — the right choice.