Skip to main content

​
Overview

Cloudflare Tunnel (cloudflared) creates an outbound connection from your machine to Cloudflare’s edge network. It acts as a reverse proxy: external requests hit Cloudflare, Cloudflare forwards them through the tunnel to your local server. TLS terminates at the edge, and no inbound port opens on your router. For LLM use, you point cloudflared at your local Ollama or vLLM port, configure a DNS record, and your model becomes reachable at a public URL like llm.example.com. It works — but Cloudflare’s proxy layer introduces constraints that matter specifically for streaming inference traffic. Tokios uses the same outbound-only connection model but adds an LLM-aware API gateway in front. SSE streaming is handled natively, there’s no proxy timeout on generation length, and per-user API keys are built in.

​
Comparison table

​
Where Cloudflare wins

  • Free tier. Cloudflare Tunnel is free on the standard plan. For a solo developer exposing a single service, the cost is $0.
  • Massive edge network. Cloudflare operates 300+ data centers worldwide. Latency to the edge is low almost everywhere.
  • DDoS protection. Cloudflare’s network absorbs volumetric attacks by default. Your home IP is never exposed.
  • Custom domains. Point llm.yourdomain.com at your tunnel. Tokios uses api.tokios.com — you don’t get a custom hostname.
  • Works for any HTTP service. Web apps, APIs, static sites — Cloudflare Tunnel doesn’t care what’s behind it. If you’re exposing a web dashboard alongside your model, one tunnel handles both.

​
Where Tokios wins

  • Native SSE streaming. No buffering, no header configuration. Tokens stream to the client as they’re generated — the gateway is built for this traffic pattern.
  • No proxy timeout on generation length. Cloudflare’s ~100–120s read timeout cuts off long generations. A large model generating 4,000 tokens on consumer hardware can easily exceed that. Tokios has no such ceiling.
  • Built-in API keys. Per-user sk-tok-… keys scoped to specific deployments. With Cloudflare, you build auth yourself — Cloudflare Access, a middleware, or a separate API gateway.
  • Protocol translation. OpenAI and Anthropic API compatibility at the gateway level. Cloudflare forwards a raw port.
  • 2-minute setup. One install command, approve in the dashboard, create a key. Cloudflare Tunnel requires DNS record creation, cloudflared config files, and (optionally) Cloudflare Access policies.

​
The SSE buffering problem

Cloudflare’s proxy buffers response bodies by default. This is correct behavior for typical web traffic — it improves throughput for full-page responses. For SSE streaming, it’s the wrong behavior: the client sits waiting for tokens, then gets the entire response dumped at once (or times out first). Cloudflare’s documentation states that proxied HTTP traffic is buffered unless the origin sends Content-Type: text/event-stream. Some LLM backends set this header; others don’t. Even when they do, intermediate proxy layers or misconfigured backends can strip it. The result is a streaming endpoint that appears to work but actually delivers all tokens in one burst at the end — defeating the purpose of streaming entirely. Tokios’s gateway is LLM-aware. It recognizes streaming completions and flushes each token to the client immediately, regardless of header configuration on the backend.

​
The timeout problem

Cloudflare’s proxy enforces a read timeout of approximately 100–120 seconds on non-Enterprise plans. A long generation — high max_tokens, large context window, or a slower model on consumer hardware — can legitimately run past that budget. When it does, the client receives a 524 error mid-generation. The model was still working; the proxy killed the connection. Enterprise customers can extend this timeout, but that’s a significant cost jump for what is otherwise a free-tier feature. Tokios imposes no ceiling on request duration. A generation that takes 5 minutes completes in full.

​
When to choose Cloudflare Tunnel

  • Already in the Cloudflare ecosystem. Your domain’s DNS is on Cloudflare, you use Cloudflare Access for other services, and adding a tunnel is incremental work.
  • Exposing web apps alongside LLMs. A dashboard, a demo UI, and a model API all behind one tunnel with one custom domain.
  • Need custom domains. llm.yourdomain.com matters for branding or integration with existing infrastructure.
  • General-purpose reverse proxy needs. You’re exposing services beyond just LLM APIs.

​
When to choose Tokios

  • LLM-specific use. Your primary goal is exposing a model API with reliable streaming — not a general web service.
  • Need reliable streaming. SSE tokens must arrive in real time, not buffered or cut off by proxy timeouts.
  • Need API keys. Multiple users or agents need scoped credentials without building a custom auth layer.
  • Want minimal setup. 2 minutes vs 30+ minutes of DNS, config, and Access policy setup.

Tunnels and streaming

Deep dive into why SSE buffering and proxy timeouts break LLM streaming.

When to use Tokios

An honest guide to when Tokios is — and isn’t — the right choice.