Skip to main content
Granite 4 is IBM’s enterprise-focused open-weight family, tuned for instruction following, tool-calling, and Retrieval Augmented Generation (RAG). Most sizes use a hybrid Mamba-2/transformer architecture — only a handful of layers use full attention, and the rest use Mamba-2 — which keeps memory use down at long context. It ships from a 350M edge model up to a 32B mixture-of-experts (MoE) model with just 9B active parameters per token, all under Apache-2.0. Run whichever size fits your hardware locally, then pair the Tokios connector to call it from any machine, teammate, or hosted agent through https://api.tokios.com — no inbound ports, no VPN.

​
Run it locally with Ollama

Granite 4 is in the Ollama library under the granite4 slug. Pull and run the default tag:
granite4:latest currently aliases to granite4:micro — the dense 3B fallback model, not one of the hybrid MoE models. If you want the hybrid Mamba-2 architecture, request a -h tag explicitly, for example ollama run granite4:tiny-h. See the Ollama library page for the full tag list — this page covers connecting it, not every runtime detail.

​
Variants & tags

Every value below is from the Ollama library and the Hugging Face model cards. Tags and download sizes change as new quantizations ship — confirm against ollama.com/library/granite4 before you standardize on one. Sources: ollama.com/library/granite4, ollama.com/library/granite4/tags, ibm-granite/granite-4.0-h-tiny, ibm-granite/granite-4.0-h-small.
The non--h sizes (350m, 1b, 3b/micro) are plain dense transformers, kept as fallback options for runtimes where Mamba-2 kernels aren’t yet optimized. Everything tagged -h — 1b-h through 32b-a9b-h — uses the hybrid Mamba-2 architecture that Granite 4 is actually built around. If you have a choice, prefer the -h tag for the memory-efficiency benefit at long context. See how inference works for background on attention vs. state-space layers.

​
Quantization & formats

Ollama’s granite4 tags ship as GGUF, mostly Q4_K_M-class quants, with -bf16 and -q8_0 variants available for the smaller dense and hybrid sizes (for example 1b-bf16, 350m-h-q8_0). The MoE hybrid models (7b-a1b-h, 32b-a9b-h) are currently published only as the default quant on Ollama. See Quantization formats and the hardware they need for the full matrix, and what quantization is for how to read Q4_K_M-style names.

​
Hardware fit

Comfort at the Ollama default quant for a modest context length. Treat every entry as an estimate — the ground truth is the actual GGUF size plus the KV cache for your context. Granite’s Mamba-2 layers keep KV-cache growth lower than a pure-attention model of the same size, so long context costs less memory than the parameter count alone suggests. See what size model fits your GPU? for the sizing math.

​
Run it — beginner to advanced

Leave Ollama listening on the default http://127.0.0.1:11434. Swap in micro, small-h, or any other tag from the table above.

​
Capabilities

  • Tool-calling: Native. Granite 4 instruct models are tuned for function-calling — define tools with an OpenAI-style function schema, and the model returns calls wrapped in <tool_call> XML tags. Ollama and vLLM/SGLang’s granite tool-call parser handle the formatting for you.
  • Retrieval Augmented Generation (RAG): Listed as a core intended use case on every Granite 4 model card, alongside summarization, text classification, and question-answering.
  • Long context: 131,072 tokens (128K) natively across every size, dense and hybrid alike.
  • Hybrid architecture: The -h models interleave a small number of full-attention layers (4 out of 40) with Mamba-2 state-space layers. This keeps KV-cache memory from growing linearly with context the way a pure-transformer model’s does, so long-context inference is cheaper on the same hardware.
  • Mixture-of-experts: 7b-a1b-h (64 experts, ~1B active) and 32b-a9b-h (72 experts, ~9B active) trade total capacity for lower per-token compute — but you still need to fit the full parameter count in memory. See the note below.

​
Connect it with Tokios

Once Granite 4 is running locally, pair the Tokios connector so you can reach it as https://api.tokios.com from Claude Code, Cline, or any other compatible client — instead of only from localhost:11434.
1

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing to get a one-time claim code (TKS-XXXX-XXXX). Install and run the connector on the machine running Ollama:
Windows
macOS
Linux
Then approve the device on the Setup or Connectors tab.
2

Point the connector at Ollama's /v1 API

Ollama exposes an OpenAI-compatible surface at /v1. For a single-model setup, the console’s Setup wizard fills this in automatically as http://127.0.0.1:11434/v1. For multiple models, set it explicitly in tokios-connector.json — see Connector config.
3

Register a deployment

In the Models tab, register the model and choose a public deployment name, for example granite-tunnel. This name — not Ollama’s local model id — is what clients send in the model field.
4

Create an API key

In the Keys tab, click Create key. Copy the sk-tok-… key now — it’s shown only once.

​
Call it from anywhere

The model field is always your registered deployment name (granite-tunnel above), never Ollama’s local model id. See Use Claude Code and Use Cline for full client setup.

​
Caveats

  • granite4:latest is the dense micro (3B) model, not a hybrid Mamba-2 model — pull a -h tag explicitly if that’s what you want.
  • The MoE hybrid models (7b-a1b-h, 32b-a9b-h) need their full total parameter count in memory even though only a fraction is active per token — size your hardware off the download size, not the active-parameter number.
  • Hybrid Mamba-2 support is newer in some inference runtimes. If vLLM, SGLang, or another server doesn’t yet support GraniteMoeHybridForCausalLM, fall back to a dense tag (3b, 1b, 350m) or update the runtime.
  • Tool-call parsing depends on the server applying Granite’s chat template and tool-call format. Ollama handles this for you; on vLLM/SGLang, pass the granite tool-call parser or calls won’t parse correctly.

Quickstart

Pair a connector and register your first model end to end.

What size model fits your GPU?

Match Granite 4 to your available VRAM before you pull it.

Quantization & hardware

Which of Granite’s formats runs on your hardware.

Which local model for which task?

See how Granite compares to other local models for tool-calling and RAG.