Choose a local model and backend

Use this guide before you register your first deployment or when you want to compare several local models. Tokios does not choose the model for you; it gives each local model a stable deployment name that clients can call.

Start with the workflow

Workflow	Optimize for	Good first check
Coding agents	Instruction following, repository reasoning, streaming stability	Ask the agent to inspect and modify a small project
Chat apps	Low latency and helpful answers	Send short prompts and compare time to first token
SDK experiments	API compatibility and predictable responses	Run the same request from `curl` and your SDK
Shared local endpoint	Stability and clear deployment names	Register one deployment per model and test `/v1/models`

Choose a backend

Tokios forwards requests to a backend that serves an OpenAI-compatible Chat Completions API. Start with the backend you already use. If you are choosing from scratch, use the simplest option for your operating system and hardware.

Backend	Often useful when…	Default Tokios upstream base
Ollama	You want a quick CLI-first local model setup	`http://127.0.0.1:11434/v1`
LM Studio	You prefer a desktop app and visual model management	`http://127.0.0.1:1234/v1`
llama.cpp	You run GGUF models directly	`http://127.0.0.1:8080/v1`
vLLM	You need a server-oriented GPU runtime	`http://127.0.0.1:8000/v1`

See Backends for current setup examples.

Name deployments for clients

The model id your backend serves does not need to be the name your clients use. Give each Tokios deployment a short public name that describes its role.

Upstream model id	Deployment name	Why it helps
`qwen2.5-coder`	`coder-local`	Coding agents can keep one stable model setting
`gemma4:e4b`	`chat-local`	Chat clients do not need to know the backend id
`mistral-small`	`fast-draft`	You can signal the intended use in the name

Keep deployment names stable. If you move a model to a different backend or machine later, update the deployment or connector route instead of changing every client.

Compare models carefully

Change one thing at a time. Keep the same prompt, client, backend settings, and hardware when you compare two deployments. Record both speed and answer quality.

Metric	How to use it
Time to first token	Measures how quickly a request starts producing output
Tokens per second	Measures generation speed after output starts
Total task time	Matters most for coding agents and multi-step workflows
Completion quality	Captures whether the model actually did the job
Stability	Catches backend restarts, timeouts, and streaming failures

For a repeatable test flow, use Benchmark your deployment.

Get Started

Models

Concepts

Reference

Choose a local model and backend

Start with the workflow

Choose a backend

Name deployments for clients

Compare models carefully

​Start with the workflow

​Choose a backend

​Name deployments for clients

​Compare models carefully

Start with the workflow

Choose a backend

Name deployments for clients

Compare models carefully