Optimize Ollama for speed and memory

Ollama is the fastest way to get a local model running — ollama run llama3 and you are serving. Under the hood it wraps llama.cpp (and MLX on Apple Silicon), and exposes a handful of environment variables and model parameters that control how it uses your hardware. The defaults work for casual use but leave significant performance on the table. This guide covers every knob that matters and how to set it for your workload.

Last verified: 2026-10-14 — Ollama v0.33.2.

Apple Silicon uses MLX. Starting with Ollama 0.19, the default backend on Apple Silicon is Apple’s MLX framework rather than llama.cpp. MLX delivers substantially faster inference on M-series chips (see Apple Silicon and MLX optimization). Most environment variables in this guide apply to both backends; where behavior differs, it is called out explicitly.

Context window tuning

The context window (num_ctx) controls how many tokens the model can process in a single request — your prompt plus the generated response combined. It is the single biggest driver of memory usage after the model weights themselves.

How to increase the Ollama context window

Ollama defaults to a 2048-token context window. Most modern models support 8K, 32K, or even 128K tokens, but Ollama does not use the model’s maximum by default — it uses 2048 to keep memory usage predictable. Set it in 3 ways, from most specific to most global: Per request (API):

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [{"role": "user", "content": "Summarize this document..."}],
    "options": {
      "num_ctx": 32768
    }
  }'

In a Modelfile (persistent per model):

FROM llama3
PARAMETER num_ctx 32768

Then build and run the custom model:

ollama create llama3-32k -f Modelfile
ollama run llama3-32k

Globally (all models, all requests):

# Linux (systemd)
sudo systemctl edit ollama.service
# Add: Environment="OLLAMA_CONTEXT_LENGTH=32768"
sudo systemctl daemon-reload && systemctl restart ollama

Windows

[System.Environment]::SetEnvironmentVariable("OLLAMA_CONTEXT_LENGTH", "32768", "User")
# Restart Ollama

macOS

launchctl setenv OLLAMA_CONTEXT_LENGTH 32768
# Restart Ollama

Context window memory cost

The KV cache grows linearly with context length. The formula:

KV cache memory = 2 × num_layers × hidden_dim × num_ctx × precision_bytes

For a 7B model (32 layers, 4096 hidden dimension) at f16 precision:

Context length	KV cache memory
2,048 (default)	~1 GB
8,192	~4 GB
32,768	~16 GB
128,000	~64 GB

Only set num_ctx as high as your workload actually needs. A coding agent that sends 2K prompts does not benefit from a 128K context window — it just wastes 64 GB of KV cache space that could serve other requests or fit a larger model.

Flash attention

Flash attention is the closest thing to a free performance improvement Ollama offers. It reworks how the model computes attention between tokens — instead of materializing the full attention matrix in memory, it computes attention in tiled chunks that fit in fast SRAM. The result:

30-50% lower memory usage for the KV cache — the biggest win at long context lengths.
Faster inference — especially on GPU, where the tiled computation maps well to tensor cores.
Same output quality — flash attention is mathematically equivalent to standard attention, not an approximation.

How to enable flash attention in Ollama

Set the environment variable:

# Linux (systemd)
sudo systemctl edit ollama.service
# Add: Environment="OLLAMA_FLASH_ATTENTION=1"
sudo systemctl daemon-reload && systemctl restart ollama

Windows

[System.Environment]::SetEnvironmentVariable("OLLAMA_FLASH_ATTENTION", "1", "User")
# Restart Ollama

macOS

launchctl setenv OLLAMA_FLASH_ATTENTION 1
# Restart Ollama

Always enable flash attention. There is no reason to leave it off unless you hit a compatibility issue with a specific model or platform. It is the single highest-impact setting in this guide.

On Apple Silicon with the MLX backend, flash attention is enabled by default and the OLLAMA_FLASH_ATTENTION variable has no effect — MLX always uses optimized attention kernels.

GPU offload strategy

Ollama automatically offloads model layers to your GPU. When the full model fits in VRAM, every layer runs on the GPU for maximum speed. When VRAM is tight, Ollama splits layers between GPU and CPU — the GPU runs what it can, and the rest falls back to system RAM.

How Ollama decides GPU layers

Ollama’s GPU scheduling works in this order:

Full offload — if the model weights + KV cache fit in VRAM, all layers go to the GPU. This is the fastest configuration.
Partial offload — if the model does not fit entirely, Ollama puts as many layers on the GPU as VRAM allows and runs the rest on CPU. Performance degrades because every forward pass must cross the PCIe bus for CPU-resident layers.
CPU-only — if no GPU is available or VRAM is too small for even a few layers, the entire model runs on CPU.

Control GPU layers with `num_gpu`

The num_gpu parameter sets how many layers to offload to the GPU. Set it in a Modelfile or per-request:

Modelfile

FROM llama3
PARAMETER num_gpu 35

API

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [{"role": "user", "content": "Hello"}],
    "options": {
      "num_gpu": 35
    }
  }'

`num_gpu` value	Behavior
`-1`	All layers on GPU (same as setting it to the total layer count)
`0`	No GPU offload — CPU only
`1` to `N`	Exactly that many layers on GPU, rest on CPU
Not set (default)	Ollama auto-detects based on available VRAM

Leave num_gpu unset and let Ollama auto-detect in most cases. Only set it manually when you need to force a specific split — for example, to reserve VRAM for a larger context window or to keep some layers on CPU for a model that barely fits.

VRAM budgeting

Each offloaded layer consumes VRAM for both weights and KV cache. A rough estimate:

VRAM per layer ≈ (parameters_per_layer × bytes_per_weight) + (KV_cache_per_layer × num_ctx)

For a 7B model at Q4_K_M (~4.5 bits per weight, ~32 layers):

Component	Per layer	32 layers total
Weights	~0.19 GB	~6 GB
KV cache (f16, 8K ctx)	~0.13 GB	~4 GB
Total		~10 GB

Leave 1-2 GB of VRAM headroom for CUDA context and overhead.

When you increase num_ctx, the KV cache grows and may push layers out of VRAM. If you set num_ctx 32768 and Ollama suddenly offloads half the layers to CPU, the KV cache ate your VRAM budget. Either reduce num_ctx, use a smaller quantization, or enable flash attention to shrink the KV cache.

KV cache precision

The KV cache stores key and value tensors for every token in every active sequence. By default, Ollama uses f16 (16-bit floating point). You can switch to quantized precision to cut memory usage at a small quality cost. Set the cache type with the OLLAMA_KV_CACHE_TYPE environment variable:

# Linux (systemd)
sudo systemctl edit ollama.service
# Add: Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
sudo systemctl daemon-reload && systemctl restart ollama

Cache type	Memory per element	Quality	When to use
`f16` (default)	2 bytes	Baseline	Maximum quality; you have plenty of memory
`q8_0`	1 byte	Near-identical to f16	Best balance — halves KV cache memory with negligible quality loss
`q4_0`	0.5 bytes	Slight degradation	Tight memory; long context or many concurrent sequences

# Halve KV cache memory
OLLAMA_KV_CACHE_TYPE=q8_0

# Quarter KV cache memory
OLLAMA_KV_CACHE_TYPE=q4_0

Combine OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1 for the biggest memory win. Flash attention reduces the KV cache footprint by 30-50%, and q8_0 halves it again — together they can cut KV cache memory by 60-75% compared to the default.

q4_0 KV cache introduces measurable quality loss on tasks that rely on precise attention over long context — needle-in-a-haystack retrieval, multi-document summarization, and long conversations. Test your specific workload before committing to q4_0.

Thread configuration

The num_thread parameter controls how many CPU threads Ollama uses for inference. The right value depends on how much work the CPU actually does. Set it in a Modelfile or per-request:

Modelfile

FROM llama3
PARAMETER num_thread 8

API

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [{"role": "user", "content": "Hello"}],
    "options": {
      "num_thread": 8
    }
  }'

You can also set it globally with OLLAMA_NUM_THREADS:

# Linux (systemd)
sudo systemctl edit ollama.service
# Add: Environment="OLLAMA_NUM_THREADS=8"
sudo systemctl daemon-reload && systemctl restart ollama

Scenario	Thread guidance
Full GPU offload (all layers on GPU)	`num_thread 2` to `num_thread 4` — the CPU handles orchestration and tokenization, not math
Partial GPU offload (some layers on CPU)	Set `num_thread` to the number of physical cores on your CPU — not hyperthreads
CPU-only (no GPU)	Set `num_thread` to the number of physical cores

More threads is not always faster. Past the number of physical cores, threads compete for cache and memory bandwidth and can slow down inference. When in doubt, start with the physical core count and benchmark from there.

On Apple Silicon, thread tuning matters less because the GPU and CPU share unified memory and the MLX backend handles scheduling. The default usually works well.

Batch size tuning

The num_batch parameter controls how many tokens Ollama processes at once during the prefill phase (prompt ingestion). The default is 512.

Modelfile

FROM llama3
PARAMETER num_batch 1024

Setting	Effect
`num_batch 512` (default)	Balanced for interactive chat with short prompts
`num_batch 1024`	Faster prefill for long prompts — roughly 50-100% faster time-to-first-token
`num_batch 2048`	Maximum throughput for document ingestion — up to 150% faster prefill
`num_batch 2048+`	Diminishing returns; higher memory usage without proportional speed gains

For interactive chat with short prompts (under 1K tokens), the default num_batch 512 is fine. Increase it only if you regularly ingest long documents and want faster time-to-first-token.

Larger batch sizes consume more VRAM during prompt ingestion. If you increase num_batch and hit out-of-memory errors, reduce it or lower num_ctx.

Parallel requests

OLLAMA_NUM_PARALLEL controls how many requests a single model processes concurrently. The default is 1 — Ollama handles one request at a time per loaded model.

# Allow 4 parallel requests per model
OLLAMA_NUM_PARALLEL=4

Value	Behavior
`1` (default)	One request at a time — lowest latency per request
`2` to `4`	Multiple requests share the GPU — higher throughput, slightly higher per-request latency
`8+`	Maximum throughput for batch workloads — requires significant VRAM for KV cache

RAM and VRAM scale with OLLAMA_NUM_PARALLEL. Each parallel request needs its own KV cache. If you serve 4 parallel requests with num_ctx 8192, you need 4× the KV cache memory compared to a single request.

When to increase parallel requests

Increase OLLAMA_NUM_PARALLEL when:

You serve multiple users or agents from the same model.
You run batch workloads where total throughput matters more than individual request latency.
You have VRAM headroom and the GPU is underutilized with single requests.

Keep it at 1 when:

You serve a single user or agent.
You prioritize low latency over throughput.
VRAM is tight.

Model memory management

Keep models loaded with `OLLAMA_KEEP_ALIVE`

By default, Ollama unloads a model from memory after 5 minutes of inactivity. The next request must reload the model from disk, which takes seconds to minutes depending on model size and disk speed. Set OLLAMA_KEEP_ALIVE to control how long models stay loaded:

# Keep models loaded indefinitely
OLLAMA_KEEP_ALIVE=-1

# Keep models loaded for 1 hour
OLLAMA_KEEP_ALIVE=1h

# Unload models immediately after each request
OLLAMA_KEEP_ALIVE=0

Value	Behavior
`5m` (default)	Unload after 5 minutes of inactivity
`-1`	Keep loaded indefinitely — no reload penalty
`0`	Unload immediately after each request
Duration (`1h`, `30m`, `24h`)	Keep loaded for that duration after the last request

For a dedicated inference server that runs the same model all day, set OLLAMA_KEEP_ALIVE=-1. The model stays in memory and every request gets instant response with no cold-start penalty.

Limit loaded models with `OLLAMA_MAX_LOADED_MODELS`

Ollama can keep multiple models loaded simultaneously. The default is 3 × (number of GPUs) or 3 for CPU-only. If you serve many models and want to limit memory usage, lower this:

# Only keep 1 model loaded at a time
OLLAMA_MAX_LOADED_MODELS=1

When the limit is reached and a new model is requested, Ollama unloads the least-recently-used model to make room.

Apple Silicon and MLX optimization

Starting with Ollama 0.19, the default backend on Apple Silicon is Apple’s MLX framework. MLX is optimized for Apple’s unified memory architecture and delivers substantially faster inference than llama.cpp on M-series chips.

Performance gains

Benchmarks from Ollama’s June 2026 MLX optimization update:

Model	Quantization	Tokens/sec (MLX)
Gemma 4 12B	NVFP4	~55 tok/s
Gemma 4 12B	Q4_K_M	~46 tok/s
Llama 3.1 8B	Q4_K_M	~70-80 tok/s

MLX also supports NVFP4 quantization on Apple Silicon, which delivers near-BF16 quality with 4-bit memory footprint — half the quality loss of Q4 quantization.

Apple Silicon optimization tips

Use MLX-optimized models. When you pull a model on Apple Silicon, Ollama automatically uses the MLX backend. No configuration needed.
Flash attention is always on. MLX uses optimized attention kernels by default. The OLLAMA_FLASH_ATTENTION variable has no effect.
Unified memory means full GPU offload. Apple Silicon’s GPU can access the full RAM pool, so -ngl 999 (all layers on GPU) works as long as the model and KV cache fit in unified memory.
Use OLLAMA_KV_CACHE_TYPE=q8_0 to stretch your memory budget. Halving the KV cache lets you run larger models or longer contexts.
Set OLLAMA_KEEP_ALIVE=-1 for always-on inference. Models stay in unified memory with no reload penalty.

If you need cross-platform portability or want to run the same configuration on NVIDIA GPU and Mac, you can force Ollama to use the llama.cpp backend on Apple Silicon. Contact Ollama support or check their documentation for the environment variable to override the backend selection.

Example configurations

Mac Studio M2 Ultra — 128 GB unified memory

# Environment variables
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KEEP_ALIVE=-1
OLLAMA_NUM_PARALLEL=4

Modelfile

FROM llama3.1:70b-q4_K_M
PARAMETER num_ctx 32768
PARAMETER num_batch 1024

This configuration runs a 70B Q4 model with a 32K context window. The 128 GB unified memory pool handles the ~40 GB model weights, ~16 GB KV cache at q8_0, and leaves headroom for 4 parallel requests. MLX delivers maximum throughput on Apple Silicon.

NVIDIA RTX 4090 — 24 GB VRAM

# Environment variables
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KEEP_ALIVE=-1
OLLAMA_NUM_PARALLEL=2
OLLAMA_NUM_THREADS=4

Modelfile

FROM llama3.1:8b-q4_K_M
PARAMETER num_ctx 16384
PARAMETER num_batch 1024

This configuration runs an 8B Q4 model with a 16K context window. The 24 GB VRAM handles the ~5 GB model weights, ~4 GB KV cache at q8_0 with flash attention, and leaves headroom for 2 parallel requests. The 4 CPU threads handle orchestration and tokenization.

Multi-GPU server — 4× RTX 4090 (96 GB total VRAM)

# Environment variables
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KEEP_ALIVE=-1
OLLAMA_NUM_PARALLEL=8
OLLAMA_NUM_THREADS=8

Modelfile

FROM llama3.1:70b-q4_K_M
PARAMETER num_ctx 8192
PARAMETER num_batch 2048

This configuration runs a 70B Q4 model across 4 GPUs. Ollama automatically distributes layers across all available GPUs. The 96 GB total VRAM handles the ~40 GB model weights, ~16 GB KV cache, and leaves headroom for 8 parallel requests. The large batch size maximizes throughput for batch workloads.

CPU-only server — 64 GB RAM, no GPU

# Environment variables
OLLAMA_KEEP_ALIVE=-1
OLLAMA_NUM_PARALLEL=1
OLLAMA_NUM_THREADS=16

Modelfile

FROM llama3.1:8b-q4_K_M
PARAMETER num_ctx 8192
PARAMETER num_batch 512
PARAMETER num_gpu 0

This configuration runs an 8B Q4 model entirely on CPU. The 16 threads match the physical core count. The small batch size and single parallel request keep memory usage manageable. Expect 5-10 tok/s — usable for interactive chat but not batch workloads.

High-throughput batch processing — RTX 4090

# Environment variables
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q4_0
OLLAMA_KEEP_ALIVE=-1
OLLAMA_NUM_PARALLEL=8
OLLAMA_NUM_THREADS=4

Modelfile

FROM llama3.1:8b-q4_K_M
PARAMETER num_ctx 4096
PARAMETER num_batch 2048

This configuration maximizes throughput for offline batch workloads. The q4_0 KV cache quarters memory usage, allowing 8 parallel requests in 24 GB VRAM. The large batch size accelerates prompt ingestion. The short 4K context keeps KV cache small. Expect 200-300 total tokens per second across all parallel requests.

Low-latency interactive serving — RTX 4090

# Environment variables
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KEEP_ALIVE=-1
OLLAMA_NUM_PARALLEL=1
OLLAMA_NUM_THREADS=4

Modelfile

FROM llama3.1:8b-q4_K_M
PARAMETER num_ctx 8192
PARAMETER num_batch 512

This configuration minimizes latency for interactive chat. Single parallel request, q8_0 KV cache for quality, and moderate batch size. Expect time-to-first-token under 500ms and 50-70 tok/s decode speed.

Common mistakes and how to avoid them

Mistake 1: Setting num_ctx too high

Problem: You set num_ctx 128000 “just in case,” but your prompts are under 2K tokens. You just wasted 64 GB of KV cache memory. Fix: Set num_ctx to match your actual workload. If your prompts are under 4K tokens, use num_ctx 4096 or num_ctx 8192. You can always increase it later for specific requests via the API.

Mistake 2: Forgetting flash attention

Problem: You are using 30-50% more VRAM than necessary because flash attention is not enabled. Fix: Always set OLLAMA_FLASH_ATTENTION=1. There is no downside on supported hardware.

Mistake 3: Not controlling model unload

Problem: You restart your request after a 10-minute break, and the first request takes 30 seconds because Ollama reloaded the model from disk. Fix: Set OLLAMA_KEEP_ALIVE=-1 for always-on inference. The model stays in memory and every request gets instant response.

Mistake 4: Using too many threads on GPU

Problem: You set num_thread 32 on a 32-thread CPU, but all layers are on the GPU. The extra threads compete for cache and slow down orchestration. Fix: When all layers are on the GPU, use num_thread 2 to num_thread 4. The CPU only handles orchestration and tokenization, not inference math.

Mistake 5: Ignoring KV cache precision

Problem: You are running out of VRAM and Ollama offloads layers to CPU, but you never considered that the KV cache is using 2× more memory than necessary. Fix: Set OLLAMA_KV_CACHE_TYPE=q8_0 to halve KV cache memory. This is the best balance of quality and memory savings.

Mistake 6: Setting OLLAMA_NUM_PARALLEL too high

Problem: You set OLLAMA_NUM_PARALLEL=16 on a 24 GB GPU, and now each request has only 1.5 GB of KV cache. Requests fail or quality degrades. Fix: Calculate your KV cache budget: (VRAM - model_weights) / num_parallel. If that leaves less than 1 GB per request, reduce num_parallel or lower num_ctx.

Troubleshooting

Ollama is slow after startup

Cause: The model was unloaded and had to reload from disk. Fix: Set OLLAMA_KEEP_ALIVE=-1 to keep the model loaded indefinitely.

Out of memory errors

Cause: The model + KV cache exceeds available VRAM or RAM. Fix: Reduce num_ctx, enable OLLAMA_FLASH_ATTENTION=1, set OLLAMA_KV_CACHE_TYPE=q8_0, or use a smaller quantization.

GPU not being used

Cause: Ollama did not detect the GPU, or VRAM is too small for even a few layers. Fix: Check ollama ps to see if layers are on GPU. For NVIDIA GPUs, ensure the NVIDIA drivers and CUDA toolkit are installed. For AMD GPUs, ensure ROCm is installed.

Slow CPU inference

Cause: Too many threads, or layers split between GPU and CPU. Fix: Set num_thread to the physical core count. If possible, offload more layers to GPU with num_gpu. Use a smaller quantization (Q4_K_M or Q5_K_M) to fit more layers on GPU.

Next steps

Ollama remote access

Expose your Ollama model to teammates and agents through Tokios — without opening ports or setting OLLAMA_HOST=0.0.0.0.

Hardware reference

Compare hardware platforms for local LLM inference — specs, model capacity, and recommendations.

Model fit finder

Find which models run on your hardware, or which hardware can run your target model.

Test your deployment

Measure latency and throughput after tuning to confirm the improvement.

Frequently asked questions

What is the best Ollama configuration for fast inference?

Enable flash attention (OLLAMA_FLASH_ATTENTION=1), set the context window to what your task actually needs (not the model maximum), use quantized KV cache (OLLAMA_KV_CACHE_TYPE=q8_0 or q4_0) for long contexts, and make sure your model is fully GPU-offloaded. See the sections above for the exact commands.

How do I reduce Ollama memory usage?

The 3 biggest levers: (1) lower num_ctx — memory scales linearly with context length, (2) quantize the KV cache with OLLAMA_KV_CACHE_TYPE=q4_0 for up to 75% KV savings, (3) use a smaller quantization of the model weights (e.g., Q4_0 instead of Q8_0). Flash attention also saves 30-50% on attention memory.

Does Ollama use MLX on Mac?

Yes — starting with Ollama 0.19, the default backend on Apple Silicon is Apple’s MLX framework, not llama.cpp. MLX delivers substantially faster inference on M-series chips (55-80 tok/s on M2 Ultra vs 25-40 tok/s with llama.cpp). The OLLAMA_FLASH_ATTENTION variable has no effect on MLX because MLX always uses optimized attention kernels.

How many parallel requests can Ollama handle?

Set OLLAMA_NUM_PARALLEL (default 1). Each parallel slot adds roughly 20-30% memory overhead for KV cache. For a 7B model at Q4 with 8K context, 4 parallel slots fit comfortably in 8 GB of VRAM. For larger models or longer contexts, start with 2 and measure.

Why is Ollama slow to load my model?

Model load time is dominated by reading weights from disk and moving them to GPU. Use an NVMe SSD (not HDD), set OLLAMA_KEEP_ALIVE to keep the model warm between requests, and set OLLAMA_MAX_LOADED_MODELS to avoid evicting and reloading. On Apple Silicon, NVFP4 models load faster because the weights are smaller.

What GPU offload settings should I use?

By default, Ollama auto-selects how many layers to offload to GPU. For maximum speed, set num_gpu to the model’s total layer count (fully offload). If you run out of VRAM, reduce num_ctx before reducing num_gpu — keeping the model fully on GPU is almost always faster than partial offload.

Sources

Ollama FAQ — official documentation on environment variables and configuration
Ollama Modelfile reference — parameter syntax and valid values
Ollama MLX performance update — June 2026 benchmarks and optimizations for Apple Silicon
Ollama GPU scheduling deep dive — VRAM management and multi-GPU behavior
Ollama performance tuning: batching, KV cache, and OOM — num_batch benchmarks and memory calculations
Ollama environment variables: complete reference — audited against v0.32.5 source

After changing Ollama configuration, re-register your Tokios deployment so the connector picks up the new upstream endpoint. See Register Model.

​Context window tuning

​How to increase the Ollama context window

​Context window memory cost

​Flash attention

​How to enable flash attention in Ollama

​GPU offload strategy

​How Ollama decides GPU layers

​Control GPU layers with num_gpu

​VRAM budgeting

​KV cache precision

​Thread configuration

​Batch size tuning

​Parallel requests

​When to increase parallel requests

​Model memory management

​Keep models loaded with OLLAMA_KEEP_ALIVE

​Limit loaded models with OLLAMA_MAX_LOADED_MODELS

​Apple Silicon and MLX optimization

​Performance gains

​Apple Silicon optimization tips

​Example configurations

​Mac Studio M2 Ultra — 128 GB unified memory

​NVIDIA RTX 4090 — 24 GB VRAM

​Multi-GPU server — 4× RTX 4090 (96 GB total VRAM)

​CPU-only server — 64 GB RAM, no GPU

​High-throughput batch processing — RTX 4090

​Low-latency interactive serving — RTX 4090

​Common mistakes and how to avoid them

​Mistake 1: Setting num_ctx too high

​Mistake 2: Forgetting flash attention

​Mistake 3: Not controlling model unload

​Mistake 4: Using too many threads on GPU

​Mistake 5: Ignoring KV cache precision

​Mistake 6: Setting OLLAMA_NUM_PARALLEL too high

​Troubleshooting

​Ollama is slow after startup

​Out of memory errors

​GPU not being used

​Slow CPU inference

​Next steps

Ollama remote access

Hardware reference

Model fit finder

Test your deployment

​Frequently asked questions

​Sources

Context window tuning

How to increase the Ollama context window

Context window memory cost

Flash attention

How to enable flash attention in Ollama

GPU offload strategy

How Ollama decides GPU layers

Control GPU layers with `num_gpu`

VRAM budgeting

KV cache precision

Thread configuration

Batch size tuning

Parallel requests

When to increase parallel requests

Model memory management

Keep models loaded with `OLLAMA_KEEP_ALIVE`

Limit loaded models with `OLLAMA_MAX_LOADED_MODELS`

Apple Silicon and MLX optimization

Performance gains

Apple Silicon optimization tips

Example configurations

Mac Studio M2 Ultra — 128 GB unified memory

NVIDIA RTX 4090 — 24 GB VRAM

Multi-GPU server — 4× RTX 4090 (96 GB total VRAM)

CPU-only server — 64 GB RAM, no GPU

High-throughput batch processing — RTX 4090

Low-latency interactive serving — RTX 4090

Common mistakes and how to avoid them

Mistake 1: Setting num_ctx too high

Mistake 2: Forgetting flash attention

Mistake 3: Not controlling model unload

Mistake 4: Using too many threads on GPU

Mistake 5: Ignoring KV cache precision

Mistake 6: Setting OLLAMA_NUM_PARALLEL too high

Troubleshooting

Ollama is slow after startup

Out of memory errors

GPU not being used

Slow CPU inference

Next steps

Frequently asked questions

Sources