Model routing strategies for local models

Tokios lets you expose multiple models — each under its own public name — and route between them by changing the model field in your request. The endpoint and API key stay the same; only the model value changes.

How routing works

Each deployment has a unique public name. When Tokios receives a request, it reads the model field, finds the deployment with that name, rewrites it to that deployment’s upstream model id, and forwards it down the connector’s tunnel. Your client config never changes — only the model value.

One connector, many models

A single connector can serve several models. List each upstream model id in Routes and map it to a named local upstream in Upstreams:

tokios-connector.json

{
  "Connector": {
    "ConnectorId": "con_...",
    "Token": "ct-...",
    "Routes": [
      { "Model": "gemma4:e4b",    "UpstreamId": "ollama" },
      { "Model": "qwen2.5-coder", "UpstreamId": "lmstudio" }
    ],
    "Upstreams": {
      "ollama":   { "BaseUrl": "http://127.0.0.1:11434/v1" },
      "lmstudio": { "BaseUrl": "http://127.0.0.1:1234/v1" }
    }
  }
}

Then register a deployment for each model in the console, giving each a public name (e.g. gemma-tunnel → gemma4:e4b, qwen-coder → qwen2.5-coder), all pointing at this connector.

Prefer to keep them separate? You can also run one connector per model, each with its own ConnectorId/Token and Upstream. Either approach works — the server only ever sends the model id down the tunnel.

Switching models in requests

With both deployments registered, switch models by changing only the model field:

# Use the Gemma model
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma-tunnel", "messages": [{"role": "user", "content": "Hello"}]}'

# Use the Qwen model
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen-coder", "messages": [{"role": "user", "content": "Hello"}]}'

The endpoint, the auth header, and every other parameter stay identical. Tokios handles the dispatch.

Swap the backend without touching clients

Because clients only reference the public name, you can move a model to different hardware or a different backend without changing any client config: update the deployment (or the connector’s upstream) to point at the new server, keep the same public name, and existing requests keep working.

Routing strategies

When you run multiple local models, sending every request to the same model wastes resources. A coding request doesn’t need a 120B reasoning model, and a simple classification task doesn’t need a 70B dense model. Routing lets you direct each request to the model best suited for the task — optimizing for speed, quality, or cost depending on the workload. Running 1 model for everything forces a trade-off. A large model handles every task well but responds slowly and consumes more VRAM. A small model responds fast but struggles with complex reasoning. Routing lets you have both:

Speed: simple queries go to a small, fast model; complex tasks go to a large model
Quality: code tasks go to a code-specialized model; general chat goes to a generalist
Cost: smaller models consume less power and VRAM, leaving headroom for demanding tasks
Resilience: if your primary model is busy or down, requests fall through to a backup

Single model per deployment

The simplest approach — register each model as its own Tokios deployment and let the client choose which to call by changing the model field.

tokios-connector.json

{
  "Connector": {
    "Routes": [
      { "Model": "qwen3-coder", "UpstreamId": "ollama" },
      { "Model": "gemma-tunnel", "UpstreamId": "ollama" }
    ],
    "Upstreams": {
      "ollama": { "BaseUrl": "http://127.0.0.1:11434/v1" }
    }
  }
}

Your client sends requests with the appropriate model value:

# Code task → code model
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-coder", "messages": [{"role": "user", "content": "Write a Python function to sort a list"}]}'

# General chat → general model
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma-tunnel", "messages": [{"role": "user", "content": "Summarize this article"}]}'

When to use: you run 2–3 models and your application controls which model to call. This is the most common starting point.

Task-based routing

Route by task type — send coding requests to a code-specialized model, chat to a generalist, and reasoning to a dense model. The client or a middleware layer inspects the request and picks the deployment.

Task type	Recommended model class	Example deployment	Why
Code generation, refactoring	Code-specialized	Qwen3-Coder-30B on vLLM	Trained specifically for code; better at syntax and logic
General chat, summarization	MoE (35B–122B)	Qwen3.5-35B-A3B on Atlas	Fast throughput; 3B active params per token
Complex reasoning, math	Dense 70B+	Qwen3-32B or 70B on vLLM	Higher quality on multi-step reasoning
Tool calling, agents	Tool-capable model	GLM-4.6 on vLLM	Strong function-calling support
Vision, multimodal	Multimodal model	Qwen3-VL on vLLM	Handles image input natively

The simplest implementation is in your application code — inspect the request type and select the deployment name:

def select_model(task_type: str) -> str:
    routing = {
        "code": "qwen3-coder",
        "chat": "gemma-tunnel",
        "reasoning": "glm-4-chat",
        "vision": "qwen3-vl",
    }
    return routing.get(task_type, "gemma-tunnel")

For more sophisticated routing, use a gateway like LiteLLM or OpenRouter that inspects requests and forwards to the correct backend based on configurable rules.

Gateway-level routing

Place a routing proxy between your clients and Tokios. The proxy decides which Tokios deployment to use based on request attributes, load, or custom rules.LiteLLM can sit in front of multiple Tokios deployments and route based on model aliases:

litellm_config.yaml

model_list:
  - model_name: gpt-4  # alias
    litellm_params:
      model: openai/qwen3-coder  # Tokios deployment
      api_base: https://api.tokios.com/v1
      api_key: sk-tok-YOUR_KEY
  - model_name: gpt-4
    litellm_params:
      model: openai/gemma-tunnel  # fallback
      api_base: https://api.tokios.com/v1
      api_key: sk-tok-YOUR_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2

Clients send requests to the LiteLLM proxy using the alias gpt-4, and the proxy routes to whichever Tokios deployment you configured.OpenRouter supports model fallback and load balancing across multiple providers. Add your Tokios endpoint as a custom provider and configure routing rules in the OpenRouter dashboard.

Fallback chains

Configure a primary model with 1 or more fallback models. If the primary is busy, offline, or returns an error, the request falls through to the next model in the chain.

import requests

FALLBACK_MODELS = ["qwen3-coder", "gemma-tunnel", "qwen3-32b"]

def chat_with_fallback(prompt: str) -> str:
    for model in FALLBACK_MODELS:
        try:
            response = requests.post(
                "https://api.tokios.com/v1/chat/completions",
                headers={
                    "Authorization": "Bearer sk-tok-YOUR_KEY",
                    "Content-Type": "application/json"
                },
                json={"model": model, "messages": [{"role": "user", "content": prompt}]},
                timeout=30
            )
            response.raise_for_status()
            return response.json()["choices"][0]["message"]["content"]
        except (requests.RequestException, KeyError):
            continue
    raise RuntimeError("All models failed")

Tokios connectors report their status to the gateway. If a connector goes offline, requests to deployments on that connector return 503. Your application can catch this and retry with a different deployment, or you can configure a gateway like LiteLLM to handle the fallback automatically.

Cost-based routing

Route simple, high-volume requests to a smaller, cheaper model and reserve the large model for tasks that need it.

Request type	Route to	Reason
Classification, extraction	MoE with 3–4B active params	Fast; low compute per token
Summarization (short)	7B–14B dense or MoE	Adequate quality; fast response
Complex reasoning	32B+ dense or 122B MoE	Higher quality on multi-step tasks
Code generation	Code-specialized model	Better output for the token budget

This strategy reduces total compute and power consumption — important when running on consumer hardware where VRAM and wattage are limited.

Decision matrix

Strategy	Complexity	Best for	When to use
Single model per deployment	Low	Getting started, 2–3 models	You control which model to call in application code
Task-based routing	Medium	Coding + chat + reasoning	You need different models for different task types
Gateway-level routing	High	Multiple clients, shared infrastructure	You want centralized routing rules without changing client code
Fallback chains	Medium	High availability	Your primary model may go offline or hit concurrency limits
Cost-based routing	Medium	High-volume, mixed workloads	You want to minimize compute cost by sending simple tasks to smaller models

These strategies are not mutually exclusive. A common production pattern combines task-based routing (code → code model) with fallback chains (primary → backup) behind a gateway proxy.

Next steps

Choose a local model

Compare models and backends to decide which deployments to register.

Benchmark your deployment

Measure throughput and latency to validate your routing decisions under real workloads.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Model routing strategies for local models

How routing works

One connector, many models

Switching models in requests

Swap the backend without touching clients

Routing strategies

Decision matrix

Next steps

Choose a local model

Benchmark your deployment

​How routing works

​One connector, many models

​Switching models in requests

​Swap the backend without touching clients

​Routing strategies

​Decision matrix

​Next steps

Choose a local model

Benchmark your deployment

How routing works

One connector, many models

Switching models in requests

Swap the backend without touching clients

Routing strategies

Decision matrix

Next steps