Run Mistral locally and use it from anywhere

Mistral AI publishes several open-weight families under Apache-2.0, each aimed at a different job: Mistral Small for general chat, Devstral for agentic coding, Magistral for reasoning, and Ministral for edge deployment. All of them run locally. Pick the tier that matches your task and hardware, then pair the Tokios connector to call it from any machine, teammate, or hosted agent through https://api.tokios.com — no inbound ports, no VPN.

Run it locally with Ollama

Each family has its own Ollama library entry. Pull the one that matches your task:

ollama run mistral-small3.2:24b   # general chat
ollama run devstral-small-2:24b   # agentic coding
ollama run magistral:24b          # reasoning
ollama run ministral-3:3b         # edge / low-VRAM

See the Ollama library for the full tag list — this page covers connecting a model, not every runtime detail.

Ollama’s default context window is small (2048 tokens) — far below any of these models’ native context. Raise num_ctx in your Modelfile or client request to use more of it.

Variants & tags

Every value below is from the model card or the Ollama library. Tags and download sizes change as new quantizations ship — confirm against the linked library page before you standardize on one.

Role	Model	Ollama tag	Context window	License
General chat	Mistral Small 3.2 (2506)	`mistral-small3.2:24b`	131,072 (128K)	Apache-2.0
General chat (larger, dense)	Mistral Large (2411)	`mistral-large:123b`	128K	Apache-2.0
Agentic coding	Devstral Small 2 (2512)	`devstral-small-2:24b`	393,216 architecture max; 256K recommended	Apache-2.0
Agentic coding (previous gen)	Devstral Small 1.0 (2505)	`devstral:24b`	131,072 (128K)	Apache-2.0
Reasoning	Magistral Small (2509)	`magistral:24b`	128K (best quality ≤40K)	Apache-2.0
Edge / low-VRAM	Ministral 3 (2512)	`ministral-3:3b`, `ministral-3:8b`, `ministral-3:14b`	262,144 (256K)	Apache-2.0
Classic 7B	Mistral 0.3	`mistral:7b`	32K	Apache-2.0

Sources: ollama.com/library/mistral-small3.2, ollama.com/library/mistral-large, ollama.com/library/devstral-small-2, ollama.com/library/devstral, ollama.com/library/magistral, ollama.com/library/ministral-3, ollama.com/library/mistral, and the corresponding Hugging Face model cards (Mistral-Small-3.2-24B-Instruct-2506, Devstral-Small-2-24B-Instruct-2512, Magistral-Small-2509, Ministral-3-3B-Instruct-2512).

Mistral also publishes a newer unified flagship, Mistral Small 4 (2603), which merges the Instruct, Reasoning, and Devstral lines into one 119B mixture-of-experts model (128 experts, 4 active; 256K context; toggle reasoning per-request with reasoning_effort). As of this writing it isn’t in the Ollama library — run it via vLLM/SGLang from mistralai/Mistral-Small-4-119B-2603 or a community GGUF, and size your hardware off all 119B parameters even though only 4 experts activate per token.

Devstral Small 2’s config lists a 393,216-token (384K) architecture maximum, but Mistral’s own model card recommends operating at 256K — treat 256K as the practical ceiling, not the raw config value.

Quantization & formats

Ollama’s default tags are GGUF Q4_K_M, with q4, q8, and fp16 variants published alongside for most of these libraries. Which format you can run depends on your hardware — see Quantization formats and the hardware they need for the full matrix, and what quantization is for how to read the Q4_K_M-style names.

Hardware fit

Comfort at roughly Q4 for a modest context length. Treat every entry as an estimate — the ground truth is the actual GGUF size plus the KV cache for your context. See What size model fits your GPU? for the sizing math.

Target	Ministral 3 (`3b`–`14b`)	Mistral Small 3.2 / Devstral / Magistral (`24b`)	Mistral Large (`123b`)
Laptop / CPU only	Comfortable at `3b`	Won’t fit at usable speed	Won’t fit
RTX 3090 / 4090 (24 GB)	Comfortable	Comfortable at `Q4`, tight at `Q8`	Won’t fit
RTX 5090 (32 GB)	Comfortable	Comfortable, more context headroom	Won’t fit
RTX 6000 Ada / A6000 (48 GB)	Comfortable	Comfortable, long context	Tight at `Q4`, short context
Apple Silicon (36 GB+ unified)	Comfortable	Comfortable	Needs 96 GB+ unified memory
AMD Strix Halo (ROCm/Vulkan, ≤96 GB)	Comfortable	Comfortable	Comfortable at `Q4`

Run it — beginner to advanced

Ollama (easiest)
LM Studio (GUI)
vLLM (advanced)
SGLang (advanced)

ollama run mistral-small3.2:24b

Leave Ollama listening on the default http://127.0.0.1:11434. Raise num_ctx to use more context; swap the tag for devstral-small-2:24b, magistral:24b, or ministral-3:3b depending on the job.

Search for Mistral, Devstral, Magistral, or Ministral in LM Studio’s model catalog, download a GGUF quant that fits your VRAM (start with Q4_K_M), then start LM Studio’s local server. It exposes an OpenAI-compatible API on http://127.0.0.1:1234/v1.

Mistral’s chat template and tool-calling need explicit flags on vLLM. For Devstral Small 2, for example:

vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --tool-call-parser mistral \
  --enable-auto-tool-choice

Magistral and Mistral Small 4 additionally need --reasoning-parser mistral to separate reasoning content from the final answer. Add --tensor-parallel-size <N> to shard across GPUs. Flags are from each model’s card — confirm against the specific model you run before deploying.

python -m sglang.launch_server \
  --model-path mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral

Add --tp <N> for tensor parallelism. Flags are from the model card’s SGLang section.

Capabilities

Native tool-calling: Mistral’s whole current lineup — Small 3.2, Devstral Small 2, Magistral, Ministral 3 — supports function calling with its own call format. Ollama applies the template for you; on vLLM/SGLang set --tool-call-parser mistral (add --enable-auto-tool-choice on vLLM) or tool calls won’t parse.
Reasoning: Magistral Small (2509) wraps its reasoning trace in [THINK]…[/THINK] tokens and needs --reasoning-parser mistral on vLLM/SGLang to separate it from the final answer. Mistral’s own guidance notes quality can degrade past 40K tokens even though the model accepts up to 128K.
Agentic coding: Devstral Small 2 is purpose-built for coding agents — repository exploration, multi-file edits, and tool use — and is validated against agent harnesses like OpenHands and Mistral’s own mistral-vibe CLI.
Edge deployment: Ministral 3 targets low-VRAM and embedded use; the 3B tier can run in roughly 8 GB of VRAM at FP8 per Mistral’s own model card, with 8B and 14B tiers available if you have more headroom.
Vision: Mistral Small 3.2, Devstral Small 2, and Magistral Small all ship with a vision encoder and accept image input alongside text.

Connect it with Tokios

Once your chosen Mistral model is running locally, pair the Tokios connector so you can reach it as https://api.tokios.com from Claude Code, Cline, or any other compatible client — instead of only from localhost:11434.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing to get a one-time claim code (TKS-XXXX-XXXX). Install and run the connector on the machine running Ollama:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Then approve the device on the Setup or Connectors tab.

Point the connector at Ollama's /v1 API

Ollama exposes an OpenAI-compatible surface at /v1. For a single-model setup, the console’s Setup wizard fills this in automatically as http://127.0.0.1:11434/v1. For multiple models — say, mistral-small3.2 for chat and devstral-small-2 for coding, both tunneled at once — set it explicitly in tokios-connector.json. See Connector config.

In the Models tab, register the model and choose a public deployment name, for example mistral-tunnel. This name — not Ollama’s local model id — is what clients send in the model field. Tokios gives whatever you’re already running locally a stable, authenticated endpoint; it doesn’t pick, host, or fine-tune the model for you.

Create an API key

In the Keys tab, click Create key. Copy the sk-tok-… key now — it’s shown only once.

Call it from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model mistral-tunnel

The model field is always your registered deployment name (mistral-tunnel above), never Ollama’s local model id. Claude Code’s model picker only lists claude* ids, so pass --model mistral-tunnel explicitly.

Caveats

Pick the tier for the job, not just the family name. “Mistral” now spans four purpose-built lines — Small (chat), Devstral (coding), Magistral (reasoning), Ministral (edge) — with different context windows and tool-calling behavior. Match the tag to the task; see Which local model for which task?.
Tool-call behavior depends on the runtime applying Mistral’s parser. Ollama handles this for you; on vLLM/SGLang, set --tool-call-parser mistral (and --enable-auto-tool-choice on vLLM) or tool calls won’t parse correctly.
Devstral Small 2’s context is runtime-dependent. The architecture supports up to 393,216 tokens, but Mistral recommends 256K as the practical operating ceiling — don’t assume the full config value is safe for quality.
Magistral’s reasoning quality can drop off well before its stated context limit. Mistral’s own card recommends keeping reasoning-heavy prompts under roughly 40K tokens even though the model accepts up to 128K.
Mistral Small 4 (2603) is the newest unified flagship but isn’t in the Ollama library as of this writing — confirm current availability before assuming an ollama run shortcut exists for it.

Quickstart

Pair a connector and register your first model end to end.

What size model fits your GPU?

Match a Mistral tier to your available VRAM before you pull it.

Quantization & hardware

Which formats — GGUF, FP8, quantized — run on your hardware.

Which local model for which task?

Compare Mistral’s chat, coding, reasoning, and edge tiers against other local models.

Get Started

Models

Concepts

Reference

Run Mistral locally and use it from anywhere

Run it locally with Ollama

Variants & tags

Quantization & formats

Hardware fit

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

​Run it locally with Ollama

​Variants & tags

​Quantization & formats

​Hardware fit

​Run it — beginner to advanced

​Capabilities

​Connect it with Tokios

​Call it from anywhere

​Caveats

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

Run it locally with Ollama

Variants & tags

Quantization & formats

Hardware fit

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats