Run Gemma 4 locally and use it from anywhere

Gemma is Google DeepMind’s open-weight model family. Gemma 4 is the current generation — multimodal (text, image, and on the smallest sizes, audio), multilingual across 140+ languages, and available as both dense and mixture-of-experts (MoE) models from edge-sized to desktop-GPU-sized. Gemma 3 is the prior generation and still widely used, especially at the small end (270m, 1b) for on-device and CPU-only work. Run whichever fits your hardware locally, then pair the Tokios connector to call it from any machine, teammate, or hosted agent through https://api.tokios.com — no inbound ports, no VPN.

Run it locally with Ollama

Both generations are in the Ollama library. For Gemma 4, pull and run the default tag (the E4B edge model):

ollama run gemma4

For the desktop-GPU tier, run ollama run gemma4:12b, ollama run gemma4:26b (MoE), or ollama run gemma4:31b. For the prior generation, ollama run gemma3 runs the 4b default — see the gemma4 and gemma3 library pages for the full tag lists — this page covers connecting it, not every runtime detail.

Ollama’s default context window is small (2048 tokens) — far below Gemma’s native 128K–256K. Raise num_ctx in your Modelfile or client request to use more of its context.

Variants & tags

Every value below is from the model cards and the Ollama library. Tags and download sizes change as new quantizations ship — confirm against ollama.com/library/gemma4 and ollama.com/library/gemma3 before you standardize on one.

Property	`E2B`	`E4B`	`12B`	`26B-A4B` (MoE)	`31B`
Ollama tag	`gemma4:e2b`	`gemma4:e4b` (alias `:latest`)	`gemma4:12b`	`gemma4:26b`	`gemma4:31b`
Download size (`Q4_K_M`)	~7.2 GB	~9.6 GB	~7.6 GB	~18 GB	~20 GB
Parameters	Edge-sized dense	Edge-sized dense	12B dense	26B total / ~4B active (128 experts, 8 routed + 1 shared)	31B dense (31.3B)
Context window	131,072	131,072	262,144	262,144	262,144
Modality	Text + image + audio (ASR)	Text + image + audio (ASR)	Text + image	Text + image	Text + image
License	Apache-2.0	Apache-2.0	Apache-2.0	Apache-2.0	Apache-2.0
Hugging Face	`google/gemma-4-E2B-it`	`google/gemma-4-E4B-it`	`google/gemma-4-12B-it`	`google/gemma-4-26B-A4B-it`	`google/gemma-4-31B-it`

Property	`270m`	`1b`	`4b`	`12b`	`27b`
Ollama tag	`gemma3:270m`	`gemma3:1b`	`gemma3:4b`	`gemma3:12b`	`gemma3:27b`
Download size (`Q4_K_M`)	— (fp16/q8_0 only)	~1.0 GB (`-it-qat`)	~3.3 GB	~8.1 GB	~17 GB
Modality	Text only	Text only	Text + image	Text + image	Text + image
License	Gemma license	Gemma license	Gemma license	Gemma license	Gemma license

Sources: ollama.com/library/gemma4, ollama.com/library/gemma3, google/gemma-4-31B-it, google/gemma-4-26B-A4B-it, google/gemma-4-E4B-it, google/gemma-4-12B-it.

The 26B-A4B is a mixture-of-experts model with only about 4B active parameters per token (8 of 128 experts routed, plus 1 shared expert), so it decodes much faster than its 26B total suggests — but all 26B parameters still have to fit in memory. Don’t size your GPU off the active-parameter count alone. See how inference works for why.

Quantization & formats

Ollama’s default tags are GGUF Q4_K_M. The gemma4 library also publishes -it-qat (quantization-aware-trained int4), -q8_0, -bf16, -mlx (Apple Silicon), -mxfp8, and -nvfp4 (NVIDIA Blackwell) variants for each size. Which format you can run depends on your hardware — see Quantization formats and the hardware they need for the full matrix, and what quantization is for how to read the Q4_K_M-style names.

QAT (quantization-aware training). Gemma 4’s -it-qat tags are int4 checkpoints Google trained to stay accurate at 4-bit, so they hold quality better than a plain post-training Q4 quant — at a comparable or smaller download: gemma4:12b-it-qat is ~7.2 GB (vs ~7.6 GB for Q4_K_M), and gemma4:e4b-it-qat is ~6.1 GB (vs ~9.6 GB). If you’re running Gemma 4 at 4-bit, prefer the -it-qat tag. See quantization-aware training for how it works.

Hardware requirements — will Gemma 4 fit your GPU?

Comfort at roughly Q4 for a modest context length. Treat every entry as an estimate — the ground truth is the actual GGUF size plus the KV cache for your context. See what size model fits your GPU? for the sizing math.

Target	`E2B`/`E4B` edge (~7–10 GB)	`12B` dense (~7.6 GB)	`26B-A4B` MoE (~18 GB)	`31B` dense (~20 GB)
Laptop / CPU-only, ≤16 GB RAM	Comfortable (`E2B`)	Tight	Won’t fit at `Q4`	Won’t fit at `Q4`
RTX 3090 / 4090 (24 GB)	Comfortable	Comfortable	Comfortable	Comfortable
RTX 5090 (32 GB, Blackwell)	Comfortable; `nvfp4` tag available	Comfortable; `nvfp4` tag available	Comfortable; `nvfp4`-style formats improving	Comfortable
RTX 6000 Ada / A6000 (48 GB)	Comfortable, long context	Comfortable, long context	Comfortable, long context	Comfortable, long context
RTX 5080 / 4080 (16 GB)	Comfortable	Comfortable	Tight — reduce context	Tight — reduce context
Apple Silicon (16 GB+ unified)	Comfortable; `mlx` tag available	Comfortable; `mlx` tag available	Needs 24 GB+ unified	Needs 24 GB+ unified

Run it — beginner to advanced

Ollama (easiest)
LM Studio (GUI)
vLLM (advanced)
SGLang (advanced)

ollama run gemma4

Leave Ollama listening on the default http://127.0.0.1:11434. Raise num_ctx to use more context, and swap in gemma4:12b, gemma4:26b, or gemma4:31b for a larger tier, or gemma3 for the prior generation.

Search for Gemma 4 (or Gemma 3) in LM Studio’s model catalog, download a GGUF quant that fits your VRAM (start with Q4_K_M, or -it-qat for a smaller int4-native download), then start LM Studio’s local server. It exposes an OpenAI-compatible API on http://127.0.0.1:1234/v1.

vllm serve google/gemma-4-12B-it \
  --max-model-len 262144

Add --tensor-parallel-size <N> to shard across GPUs. Swap in google/gemma-4-26B-A4B-it, google/gemma-4-31B-it, or an E2B/E4B repo for a different tier — check the model card for the current recommended transformers/vllm version before serving.

python -m sglang.launch_server \
  --model-path google/gemma-4-12B-it \
  --context-length 262144

Add --tp-size <N> for tensor parallelism. Check the model card for the current recommended transformers/sglang version before serving.

Capabilities

Reasoning: Gemma 4 supports a configurable thinking mode, off by default. Enable it by including the <|think|> control token at the start of the system prompt (or enable_thinking=True if you’re using the model card’s parse_response helper); the model then emits step-by-step reasoning before its final answer.
Native tool-calling: Yes, on Gemma 4 — the model card documents native function-calling for structured tool use and agentic workflows, using the standard system/user/assistant roles.
Vision: Multimodal — accepts text and images (with variable aspect ratio and resolution) across the whole Gemma 4 family; Gemma 3 (4b and up) is also vision-capable.
Audio: E2B and E4B only — native automatic speech recognition (ASR) and speech-to-translated-text, up to 30 seconds of audio per request.
Long context: 131,072 tokens natively on E2B/E4B, 262,144 tokens on 12B, 26B-A4B, and 31B.

Connect it with Tokios

Once Gemma is running locally, pair the Tokios connector so you can reach it as https://api.tokios.com from Claude Code, Cline, or any other compatible client — instead of only from localhost:11434.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing to get a one-time claim code (TKS-XXXX-XXXX). Install and run the connector on the machine running Ollama:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Then approve the device on the Setup or Connectors tab.

Point the connector at Ollama's /v1 API

Ollama exposes an OpenAI-compatible surface at /v1. For a single-model setup, the console’s Setup wizard fills this in automatically as http://127.0.0.1:11434/v1. For multiple models, set it explicitly in tokios-connector.json — see Connector config.

In the Models tab, register the model and choose a public deployment name, for example gemma-tunnel. This name — not Ollama’s local model id — is what clients send in the model field.

Create an API key

In the Keys tab, click Create key. Copy the sk-tok-… key now — it’s shown only once.

Call it from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Summarize this document and list open questions."}]
  }'

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model gemma-tunnel

The model field is always your registered deployment name (gemma-tunnel above), never Ollama’s local model id. See Use Claude Code and Use Cline for full client setup.

Caveats

Gemma 4’s thinking mode is off by default — the opposite of some other reasoning families. If you want step-by-step reasoning, you have to opt in with the <|think|> control token or your client’s equivalent.
Audio input (ASR and speech translation) only works on the E2B and E4B sizes — 12B, 26B-A4B, and 31B accept text and image only.
The 26B-A4B MoE needs all 26B parameters in memory even though only about 4B are active per token — size your hardware off the full download, not the active count.
Licensing differs by generation: Gemma 4 is Apache 2.0 and ungated, while Gemma 3 uses Google’s custom Gemma license and its Hugging Face checkpoints are gated (you accept the terms before downloading; the Ollama tags are redistributed under the same license). Google’s Gemma Prohibited Use Policy applies to both — check the license terms before commercial or sensitive deployments.
Gemma 3 remains useful for very small, CPU-friendly deployments (270m, 1b) where Gemma 4’s smallest tier (E2B) may still be heavier than you need.

Quickstart

Pair a connector and register your first model end to end.

What size model fits your GPU?

Match Gemma to your available VRAM before you pull it.

Quantization & hardware

Which of Gemma’s formats — GGUF, QAT, MLX, NVFP4 — runs on your hardware.

Which local model for which task?

See how Gemma compares to other local models by task.

Get Started

Models

Concepts

Reference

Run Gemma 4 locally and use it from anywhere

Run it locally with Ollama

Variants & tags

Quantization & formats

Hardware requirements — will Gemma 4 fit your GPU?

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

​Run it locally with Ollama

​Variants & tags

​Quantization & formats

​Hardware requirements — will Gemma 4 fit your GPU?

​Run it — beginner to advanced

​Capabilities

​Connect it with Tokios

​Call it from anywhere

​Caveats

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

Run it locally with Ollama

Variants & tags

Quantization & formats

Hardware requirements — will Gemma 4 fit your GPU?

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats