Skip to main content
A local LLM API needs two things most general-purpose tunnels don’t optimize for: token streaming (Server-Sent Events that flush as each token is generated, not buffered into one response) and long-lived connections (a generation can legitimately run past a proxy’s default timeout). Cloudflare Tunnel, ngrok, and Tailscale are solid tools for the traffic they’re designed around — but each documents constraints that surface specifically on this workload. Tokios’s connector is outbound-only like a tunnel, but built around streaming LLM traffic from the start: no read/write timeout to configure around, and keys scoped to your models instead of raw ports.

​
The two things that break

​
Streaming gets buffered

An LLM completion streamed over SSE sends partial tokens as soon as they’re generated — the client expects a steady drip, not one chunk at the end. Some reverse proxies buffer response bodies by default to optimize throughput for typical web traffic, which is the wrong behavior for SSE: the client sits waiting, then gets the whole response at once (or times out first).
Top bar shows a buffered response that stays empty then fills all at once; bottom bar shows a streamed response filling steadily token by token.

Buffered: the client waits, then the whole answer dumps at once. Streamed: tokens flush as they're generated.

​
Requests run longer than the timeout allows

A large model on modest hardware can take well past the ordinary web-request budget to finish a long generation. Any proxy or tunnel with a fixed read/write timeout will cut the connection before the model is done, and the client sees a gateway error instead of a completion — even though the model was still working.

​
Comparison

Every limit above is the vendor’s own documented behavior for its general-purpose use case, not a failure — Cloudflare Tunnel, ngrok, and Tailscale all do what they’re designed to do well. The friction shows up specifically when the workload is a streaming LLM API with generations that can run long.

​
Where Tokios fits

Tokios isn’t a replacement for Tailscale or Cloudflare Tunnel in general. Keep using Tailscale for SSH into a home server, or Cloudflare Tunnel for a low-traffic web app — those are exactly the jobs they’re built for. Tokios is specifically the connect layer for one path: getting a coding agent or API client to a streaming LLM endpoint running on your own hardware, without buffering SSE or racing a proxy timeout. The tokios-connector dials out over a single outbound WebSocket, the same direction-flipping idea behind a tunnel — see How it works for the full request flow. From there, every request authenticates with an sk-tok-… API key scoped to specific deployments, rather than a raw port or hostname being reachable at all.

​
Next steps

Quickstart

Pair a connector, register a model, and send your first streaming request.

How Tokios routes requests

See the outbound-only architecture and request flow in detail.
Serving Ollama specifically? See Ollama remote access for backend-specific setup notes.