Run Gemma 4 locally and use it from anywhere

Last verified: 2026-10-14

Gemma is Google DeepMind’s open-weight model family. Gemma 4 is the current generation — multimodal (text, image, and on the smallest sizes, audio), multilingual across 140+ languages, and available as both dense and mixture-of-experts (MoE) models from edge-sized to desktop-GPU-sized. Gemma 3 is the prior generation and still widely used, especially at the small end (270m, 1b) for on-device and CPU-only work. Run whichever fits your hardware locally, then pair the Tokios connector to call it from any machine, teammate, or hosted agent through https://api.tokios.com — no inbound ports, no VPN.

Run it locally with Ollama

Both generations are in the Ollama library. For Gemma 4, pull and run the default tag (the E4B edge model):

ollama run gemma4

For the desktop-GPU tier, run ollama run gemma4:12b, ollama run gemma4:26b (MoE), or ollama run gemma4:31b. For the prior generation, ollama run gemma3 runs the 4b default — see the gemma4 and gemma3 library pages for the full tag lists — this page covers connecting it, not every runtime detail.

Ollama’s default context window is small (2048 tokens) — far below Gemma’s native 128K–256K. Raise num_ctx in your Modelfile or client request to use more of its context.

Variants & tags

Every value below is from the model cards and the Ollama library. Tags and download sizes change as new quantizations ship — confirm against ollama.com/library/gemma4 and ollama.com/library/gemma3 before you standardize on one.

Property	`E2B`	`E4B`	`12B`	`26B-A4B` (MoE)	`31B`
Ollama tag	`gemma4:e2b`	`gemma4:e4b` (alias `:latest`)	`gemma4:12b`	`gemma4:26b`	`gemma4:31b`
Download size (`Q4_K_M`)	~7.2 GB	~9.6 GB	~7.6 GB	~18 GB	~20 GB
Parameters	Edge-sized dense	Edge-sized dense	12B dense	26B total / ~4B active (128 experts, 8 routed + 1 shared)	31B dense (31.3B)
Context window	131,072	131,072	262,144	262,144	262,144
Modality	Text + image + audio (ASR)	Text + image + audio (ASR)	Text + image	Text + image	Text + image
License	Apache-2.0	Apache-2.0	Apache-2.0	Apache-2.0	Apache-2.0
Hugging Face	`google/gemma-4-E2B-it`	`google/gemma-4-E4B-it`	`google/gemma-4-12B-it`	`google/gemma-4-26B-A4B-it`	`google/gemma-4-31B-it`

Property	`270m`	`1b`	`4b`	`12b`	`27b`
Ollama tag	`gemma3:270m`	`gemma3:1b`	`gemma3:4b`	`gemma3:12b`	`gemma3:27b`
Download size (`Q4_K_M`)	— (fp16/q8_0 only)	~1.0 GB (`-it-qat`)	~3.3 GB	~8.1 GB	~17 GB
Modality	Text only	Text only	Text + image	Text + image	Text + image
License	Gemma license	Gemma license	Gemma license	Gemma license	Gemma license

Sources: ollama.com/library/gemma4, ollama.com/library/gemma3, google/gemma-4-31B-it, google/gemma-4-26B-A4B-it, google/gemma-4-E4B-it, google/gemma-4-12B-it.

The 26B-A4B is a mixture-of-experts model with only about 4B active parameters per token (8 of 128 experts routed, plus 1 shared expert), so it decodes much faster than its 26B total suggests — but all 26B parameters still have to fit in memory. Don’t size your GPU off the active-parameter count alone. See how inference works for why.

Quantization & formats

Ollama’s default tags are GGUF Q4_K_M. The gemma4 library also publishes -it-qat (quantization-aware-trained int4), -q8_0, -bf16, -mlx (Apple Silicon), -mxfp8, and -nvfp4 (NVIDIA Blackwell) variants for each size. Which format you can run depends on your hardware — see Quantization formats and the hardware they need for the full matrix, and what quantization is for how to read the Q4_K_M-style names.

QAT (quantization-aware training). Gemma 4’s -it-qat tags are int4 checkpoints Google trained to stay accurate at 4-bit, so they hold quality better than a plain post-training Q4 quant — at a comparable or smaller download: gemma4:12b-it-qat is ~7.2 GB (vs ~7.6 GB for Q4_K_M), and gemma4:e4b-it-qat is ~6.1 GB (vs ~9.6 GB). If you’re running Gemma 4 at 4-bit, prefer the -it-qat tag. See quantization-aware training for how it works.

Apple Silicon? MLX builds of Gemma 4 are available from mlx-community on Hugging Face. Serve them with mlx_lm.server for native unified-memory inference on Mac Studio, Mac mini, and MacBook Pro — see MLX-LM Server in the backends reference.

Hardware requirements — will Gemma 4 fit your GPU?

Comfort at roughly Q4 for a modest context length. Treat every entry as an estimate — the ground truth is the actual GGUF size plus the KV cache for your context. See what size model fits your GPU? for the sizing math.

Target	`E2B`/`E4B` edge (~7–10 GB)	`12B` dense (~7.6 GB)	`26B-A4B` MoE (~18 GB)	`31B` dense (~20 GB)
Laptop / CPU-only, ≤16 GB RAM	Comfortable (`E2B`)	Tight	Won’t fit at `Q4`	Won’t fit at `Q4`
RTX 3090 / 4090 (24 GB)	Comfortable	Comfortable	Comfortable	Comfortable
RTX 5090 (32 GB, Blackwell)	Comfortable; `nvfp4` tag available	Comfortable; `nvfp4` tag available	Comfortable; `nvfp4`-style formats improving	Comfortable
RTX 6000 Ada / A6000 (48 GB)	Comfortable, long context	Comfortable, long context	Comfortable, long context	Comfortable, long context
RTX 5080 / 4080 (16 GB)	Comfortable	Comfortable	Tight — reduce context	Tight — reduce context
Apple Silicon (16 GB+ unified)	Comfortable; `mlx` tag available	Comfortable; `mlx` tag available	Needs 24 GB+ unified	Needs 24 GB+ unified

Run it — beginner to advanced

Ollama (easiest)
LM Studio (GUI)
vLLM (advanced)
SGLang (advanced)

ollama run gemma4

Leave Ollama listening on the default http://127.0.0.1:11434. Raise num_ctx to use more context, and swap in gemma4:12b, gemma4:26b, or gemma4:31b for a larger tier, or gemma3 for the prior generation.

Search for Gemma 4 (or Gemma 3) in LM Studio’s model catalog, download a GGUF quant that fits your VRAM (start with Q4_K_M, or -it-qat for a smaller int4-native download), then start LM Studio’s local server. It exposes an OpenAI-compatible API on http://127.0.0.1:1234/v1.

vllm serve google/gemma-4-12B-it \
  --max-model-len 262144

Add --tensor-parallel-size <N> to shard across GPUs. Swap in google/gemma-4-26B-A4B-it, google/gemma-4-31B-it, or an E2B/E4B repo for a different tier — check the model card for the current recommended transformers/vllm version before serving.

python -m sglang.launch_server \
  --model-path google/gemma-4-12B-it \
  --context-length 262144

Add --tp-size <N> for tensor parallelism. Check the model card for the current recommended transformers/sglang version before serving.

Capabilities

Reasoning: Gemma 4 supports a configurable thinking mode, off by default. Enable it by including the <|think|> control token at the start of the system prompt (or enable_thinking=True if you’re using the model card’s parse_response helper); the model then emits step-by-step reasoning before its final answer.
Native tool-calling: Yes, on Gemma 4 — the model card documents native function-calling for structured tool use and agentic workflows, using the standard system/user/assistant roles.
Vision: Multimodal — accepts text and images (with variable aspect ratio and resolution) across the whole Gemma 4 family; Gemma 3 (4b and up) is also vision-capable.
Audio: E2B and E4B only — native automatic speech recognition (ASR) and speech-to-translated-text, up to 30 seconds of audio per request.
Long context: 131,072 tokens natively on E2B/E4B, 262,144 tokens on 12B, 26B-A4B, and 31B.

Connect it with Tokios

Once Gemma is running locally, pair the Tokios connector so you can reach it as https://api.tokios.com from Claude Code, Cline, or any other compatible client — instead of only from localhost:11434.

Pair the Tokios connector

Sign in at tokios.com/console, open the Setup tab, and click Start pairing to get a one-time claim code (TKS-XXXX-XXXX). Install and run the connector on the machine running Ollama:

Windows

irm https://tokios.com/downloads/install/windows | iex; tokios-install -Code 'TKS-XXXX-XXXX'

macOS

curl -fsSL https://tokios.com/downloads/install/macos | sh -s -- --code TKS-XXXX-XXXX

Linux

curl -fsSL https://tokios.com/downloads/install/linux | sh -s -- --code TKS-XXXX-XXXX

Then approve the device on the Setup or Connectors tab.

Point the connector at Ollama's /v1 API

Ollama exposes an OpenAI-compatible surface at /v1. For a single-model setup, the console’s Setup wizard fills this in automatically as http://127.0.0.1:11434/v1. For multiple models, set it explicitly in tokios-connector.json — see Connector config.

In the Models tab, register the model and choose a public deployment name, for example gemma-tunnel. This name — not Ollama’s local model id — is what clients send in the model field.

Create an API key

In the Keys tab, click Create key. Copy the sk-tok-… key now — it’s shown only once.

Call it from anywhere

curl https://api.tokios.com/v1/chat/completions \
  -H "Authorization: Bearer sk-tok-YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-tunnel",
    "stream": true,
    "messages": [{"role": "user", "content": "Summarize this document and list open questions."}]
  }'

export ANTHROPIC_BASE_URL=https://api.tokios.com
export ANTHROPIC_API_KEY=sk-tok-YOUR_KEY
claude --model gemma-tunnel

The model field is always your registered deployment name (gemma-tunnel above), never Ollama’s local model id. See Use Claude Code and Use Cline for full client setup.

Caveats

Gemma 4’s thinking mode is off by default — the opposite of some other reasoning families. If you want step-by-step reasoning, you have to opt in with the <|think|> control token or your client’s equivalent.
Audio input (ASR and speech translation) only works on the E2B and E4B sizes — 12B, 26B-A4B, and 31B accept text and image only.
The 26B-A4B MoE needs all 26B parameters in memory even though only about 4B are active per token — size your hardware off the full download, not the active count.
Licensing differs by generation: Gemma 4 is Apache 2.0 and ungated, while Gemma 3 uses Google’s custom Gemma license and its Hugging Face checkpoints are gated (you accept the terms before downloading; the Ollama tags are redistributed under the same license). Google’s Gemma Prohibited Use Policy applies to both — check the license terms before commercial or sensitive deployments.
Gemma 3 remains useful for very small, CPU-friendly deployments (270m, 1b) where Gemma 4’s smallest tier (E2B) may still be heavier than you need.

New same-class alternative. Meta’s Muse Glimmer 30B lands in the same tier as gemma4:31b — dense, multimodal, Apache 2.0, and sized for one consumer GPU. Meta benchmarks them against each other directly and its numbers favour Muse Glimmer, but they are vendor-run with no independent reproduction yet. Gemma 4 still spans a much wider range, from E2B on edge hardware up to 31B; Muse Glimmer ships at one size. Test both on your own workload.

FAQ

How much VRAM does Gemma 4 need?

At Q4_K_M, the edge models (E2B, E4B) need ~7–10 GB, the 12B dense needs ~7.6 GB, the 26B-A4B MoE needs ~18 GB, and the 31B dense needs ~20 GB. The -it-qat int4-native tags are smaller still — for example gemma4:12b-it-qat is ~7.2 GB. BF16 builds are much larger: the 31B alone is ~57 GB. Add KV cache on top for your context length.

Can Gemma 4 run on a 24 GB GPU?

Yes — every Gemma 4 variant fits a 24 GB card (RTX 3090 / 4090) at Q4_K_M, including the 31B dense at ~20 GB and the 26B-A4B MoE at ~18 GB. The edge models and 12B leave plenty of headroom for long context; the 31B is comfortable but tighter at very large context lengths.

Is Gemma 4 open source?

Gemma 4 is released under Apache 2.0 and is ungated — no acceptable-use policy or sign-up to download. Gemma 3 uses Google’s custom Gemma license and its Hugging Face checkpoints are gated (you accept the terms before downloading). Google’s Gemma Prohibited Use Policy applies to both generations. Note that open weights and open source are not the same thing — the training data and pipeline are not released.

What is QAT and how does it help?

QAT (quantization-aware training) means Google trained the model to stay accurate at 4-bit from the start, rather than applying quantization after training. The result: -it-qat tags hold quality better than a plain post-training Q4_K_M quant, at a comparable or smaller download — gemma4:12b-it-qat is ~7.2 GB vs ~7.6 GB for Q4_K_M, and gemma4:e4b-it-qat is ~6.1 GB vs ~9.6 GB. If you are running Gemma 4 at 4-bit, prefer the -it-qat tag.

Should I use Gemma 3 or Gemma 4?

Use Gemma 4 — it is the current generation, with multimodal input (text + image across all sizes, audio on E2B/E4B), Apache 2.0 licensing, and context windows of 131K–262K tokens. Gemma 3 remains useful for very small, CPU-friendly deployments (270m, 1b) where Gemma 4’s smallest tier (E2B) is still heavier than you need, or if you specifically rely on a Gemma 3 tag that has no Gemma 4 equivalent.

Quickstart

Pair a connector and register your first model end to end.

What size model fits your GPU?

Match Gemma to your available VRAM before you pull it.

Quantization & hardware

Which of Gemma’s formats — GGUF, QAT, MLX, NVFP4 — runs on your hardware.

Which local model for which task?

See how Gemma compares to other local models by task.

Get Started

Models

Concepts

Reference

Run Gemma 4 locally and use it from anywhere

Run it locally with Ollama

Variants & tags

Quantization & formats

Hardware requirements — will Gemma 4 fit your GPU?

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats

FAQ

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

​Run it locally with Ollama

​Variants & tags

​Quantization & formats

​Hardware requirements — will Gemma 4 fit your GPU?

​Run it — beginner to advanced

​Capabilities

​Connect it with Tokios

​Call it from anywhere

​Caveats

​FAQ

Quickstart

What size model fits your GPU?

Quantization & hardware

Which local model for which task?

Run it locally with Ollama

Variants & tags

Quantization & formats

Hardware requirements — will Gemma 4 fit your GPU?

Run it — beginner to advanced

Capabilities

Connect it with Tokios

Call it from anywhere

Caveats

FAQ