Benchmark a Tokios deployment

Use this guide to compare two or more Tokios deployments. Keep the test simple at first: same prompt, same client, same hardware, same backend settings. Change only the deployment name in the model field.

Prerequisites

A running connector. See Connector Install.
One or more registered deployments. See Register Model.
A Tokios API key. See API Keys.
A local backend that works before Tokios is involved. See Backends.

Test a basic request

Start with a small non-streaming request. Replace the key and model with your own values.

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "coder-local",
    "messages": [
      { "role": "user", "content": "Reply with one short sentence." }
    ]
  }'

This confirms authentication, deployment routing, connector connectivity, and backend response handling.

Test streaming

Then test streaming behavior. This is closer to how many coding agents and chat tools feel in practice.

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "coder-local",
    "stream": true,
    "messages": [
      { "role": "user", "content": "Write a five item checklist for reviewing a pull request." }
    ]
  }'

If streaming stalls, check the local backend directly, then check the connector logs and client base URL.

Compare two deployments

Use the same request body and change only model.

# First deployment
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"coder-local","messages":[{"role":"user","content":"Explain this function in plain English."}]}'

# Second deployment
curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"fast-draft","messages":[{"role":"user","content":"Explain this function in plain English."}]}'

Record the result in a small table:

Deployment	Time to first output	Total time	Quality notes	Errors
`coder-local`
`fast-draft`

Test with your actual client

After curl works, test through the client you plan to use.

Client type	What to test
Claude Code	A small one-shot prompt with `--model your-deployment`
OpenAI Codex	A short repository question using the Tokios profile
SDK app	One request from the same code path your app will use
Chat UI	A short prompt and one longer streaming response

Interpret the results

Do not optimize for a single number. A deployment that streams quickly but fails coding tasks may be worse than a slower model that follows instructions reliably.

Signal	What it may mean
Slow first token	Backend loading, large prompt, cold model, or hardware limits
Fast tokens but poor answers	Model may not fit the workflow
Works in `curl` but not in a client	Base URL, API shape, or model field mismatch
Works locally but not through Tokios	Connector, deployment, or API key issue

Keep your best deployment name stable. If you later move it to a faster backend or different machine, clients can keep using the same model value.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Benchmark a Tokios deployment

Prerequisites

Test a basic request

Test streaming

Compare two deployments

Test with your actual client

Interpret the results

​Prerequisites

​Test a basic request

​Test streaming

​Compare two deployments

​Test with your actual client

​Interpret the results

Prerequisites

Test a basic request

Test streaming

Compare two deployments

Test with your actual client

Interpret the results