> ## Documentation Index > Fetch the complete documentation index at: https://docs.tokios.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel. > Use 'connector' for the local tokios-connector process. > Use 'deployment' for the public model name in the API model field. > Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend. > API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-. # Optimize Ollama for speed and memory > Tune Ollama for maximum speed — context window, GPU offload, flash attention, KV cache quantization, and parallel requests. Cut latency 2-5x with these settings. Ollama is the fastest way to get a local model running — `ollama run llama3` and you are serving. Under the hood it wraps llama.cpp (and MLX on Apple Silicon), and exposes a handful of environment variables and model parameters that control how it uses your hardware. The defaults work for casual use but leave significant performance on the table. This guide covers every knob that matters and how to set it for your workload. > [!NOTE] > > Last verified: 2026-10-14 — Ollama v0.33.2. > [!NOTE] > > **Apple Silicon uses MLX.** Starting with Ollama 0.19, the default backend on Apple Silicon is Apple's MLX framework rather than llama.cpp. MLX delivers substantially faster inference on M-series chips (see [Apple Silicon and MLX optimization](#apple-silicon-and-mlx-optimization)). Most environment variables in this guide apply to both backends; where behavior differs, it is called out explicitly. ## Context window tuning The context window (`num_ctx`) controls how many tokens the model can process in a single request — your prompt plus the generated response combined. It is the single biggest driver of memory usage after the model weights themselves. ### How to increase the Ollama context window Ollama defaults to a 2048-token context window. Most modern models support 8K, 32K, or even 128K tokens, but Ollama does not use the model's maximum by default — it uses 2048 to keep memory usage predictable. Set it in 3 ways, from most specific to most global: **Per request (API):** ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3", "messages": [{"role": "user", "content": "Summarize this document..."}], "options": { "num_ctx": 32768 } }' ``` **In a Modelfile (persistent per model):** ```dockerfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3 PARAMETER num_ctx 32768 ``` Then build and run the custom model: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} ollama create llama3-32k -f Modelfile ollama run llama3-32k ``` **Globally (all models, all requests):** ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Linux (systemd) sudo systemctl edit ollama.service # Add: Environment="OLLAMA_CONTEXT_LENGTH=32768" sudo systemctl daemon-reload && systemctl restart ollama ``` ```powershell Windows theme={"theme":{"light":"github-light","dark":"css-variables"}} [System.Environment]::SetEnvironmentVariable("OLLAMA_CONTEXT_LENGTH", "32768", "User") # Restart Ollama ``` ```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}} launchctl setenv OLLAMA_CONTEXT_LENGTH 32768 # Restart Ollama ``` ### Context window memory cost The KV cache grows linearly with context length. The formula: ``` KV cache memory = 2 × num_layers × hidden_dim × num_ctx × precision_bytes ``` For a 7B model (32 layers, 4096 hidden dimension) at f16 precision: | Context length | KV cache memory | | - | - | | 2,048 (default) | \~1 GB | | 8,192 | \~4 GB | | 32,768 | \~16 GB | | 128,000 | \~64 GB | > [!TIP] > > Only set `num_ctx` as high as your workload actually needs. A coding agent that sends 2K prompts does not benefit from a 128K context window — it just wastes 64 GB of KV cache space that could serve other requests or fit a larger model. ## Flash attention Flash attention is the closest thing to a free performance improvement Ollama offers. It reworks how the model computes attention between tokens — instead of materializing the full attention matrix in memory, it computes attention in tiled chunks that fit in fast SRAM. The result: * **30-50% lower memory usage** for the KV cache — the biggest win at long context lengths. * **Faster inference** — especially on GPU, where the tiled computation maps well to tensor cores. * **Same output quality** — flash attention is mathematically equivalent to standard attention, not an approximation. ### How to enable flash attention in Ollama Set the environment variable: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Linux (systemd) sudo systemctl edit ollama.service # Add: Environment="OLLAMA_FLASH_ATTENTION=1" sudo systemctl daemon-reload && systemctl restart ollama ``` ```powershell Windows theme={"theme":{"light":"github-light","dark":"css-variables"}} [System.Environment]::SetEnvironmentVariable("OLLAMA_FLASH_ATTENTION", "1", "User") # Restart Ollama ``` ```bash macOS theme={"theme":{"light":"github-light","dark":"css-variables"}} launchctl setenv OLLAMA_FLASH_ATTENTION 1 # Restart Ollama ``` > [!TIP] > > **Always enable flash attention.** There is no reason to leave it off unless you hit a compatibility issue with a specific model or platform. It is the single highest-impact setting in this guide. > [!NOTE] > > On Apple Silicon with the MLX backend, flash attention is enabled by default and the `OLLAMA_FLASH_ATTENTION` variable has no effect — MLX always uses optimized attention kernels. ## GPU offload strategy Ollama automatically offloads model layers to your GPU. When the full model fits in VRAM, every layer runs on the GPU for maximum speed. When VRAM is tight, Ollama splits layers between GPU and CPU — the GPU runs what it can, and the rest falls back to system RAM. ### How Ollama decides GPU layers Ollama's GPU scheduling works in this order: 1. **Full offload** — if the model weights + KV cache fit in VRAM, all layers go to the GPU. This is the fastest configuration. 2. **Partial offload** — if the model does not fit entirely, Ollama puts as many layers on the GPU as VRAM allows and runs the rest on CPU. Performance degrades because every forward pass must cross the PCIe bus for CPU-resident layers. 3. **CPU-only** — if no GPU is available or VRAM is too small for even a few layers, the entire model runs on CPU. ### Control GPU layers with `num_gpu` The `num_gpu` parameter sets how many layers to offload to the GPU. Set it in a Modelfile or per-request: ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3 PARAMETER num_gpu 35 ``` ```bash API theme={"theme":{"light":"github-light","dark":"css-variables"}} curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3", "messages": [{"role": "user", "content": "Hello"}], "options": { "num_gpu": 35 } }' ``` | `num_gpu` value | Behavior | | - | - | | `-1` | All layers on GPU (same as setting it to the total layer count) | | `0` | No GPU offload — CPU only | | `1` to `N` | Exactly that many layers on GPU, rest on CPU | | Not set (default) | Ollama auto-detects based on available VRAM | > [!TIP] > > Leave `num_gpu` unset and let Ollama auto-detect in most cases. Only set it manually when you need to force a specific split — for example, to reserve VRAM for a larger context window or to keep some layers on CPU for a model that barely fits. ### VRAM budgeting Each offloaded layer consumes VRAM for both weights and KV cache. A rough estimate: ``` VRAM per layer ≈ (parameters_per_layer × bytes_per_weight) + (KV_cache_per_layer × num_ctx) ``` For a 7B model at Q4\_K\_M (\~4.5 bits per weight, \~32 layers): | Component | Per layer | 32 layers total | | - | - | - | | Weights | \~0.19 GB | \~6 GB | | KV cache (f16, 8K ctx) | \~0.13 GB | \~4 GB | | **Total** | | **\~10 GB** | Leave 1-2 GB of VRAM headroom for CUDA context and overhead. > [!WARNING] > > When you increase `num_ctx`, the KV cache grows and may push layers out of VRAM. If you set `num_ctx 32768` and Ollama suddenly offloads half the layers to CPU, the KV cache ate your VRAM budget. Either reduce `num_ctx`, use a smaller quantization, or enable flash attention to shrink the KV cache. ## KV cache precision The KV cache stores key and value tensors for every token in every active sequence. By default, Ollama uses `f16` (16-bit floating point). You can switch to quantized precision to cut memory usage at a small quality cost. Set the cache type with the `OLLAMA_KV_CACHE_TYPE` environment variable: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Linux (systemd) sudo systemctl edit ollama.service # Add: Environment="OLLAMA_KV_CACHE_TYPE=q8_0" sudo systemctl daemon-reload && systemctl restart ollama ``` | Cache type | Memory per element | Quality | When to use | | - | - | - | - | | `f16` (default) | 2 bytes | Baseline | Maximum quality; you have plenty of memory | | `q8_0` | 1 byte | Near-identical to f16 | Best balance — halves KV cache memory with negligible quality loss | | `q4_0` | 0.5 bytes | Slight degradation | Tight memory; long context or many concurrent sequences | ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Halve KV cache memory OLLAMA_KV_CACHE_TYPE=q8_0 # Quarter KV cache memory OLLAMA_KV_CACHE_TYPE=q4_0 ``` > [!TIP] > > Combine `OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1` for the biggest memory win. Flash attention reduces the KV cache footprint by 30-50%, and q8\_0 halves it again — together they can cut KV cache memory by 60-75% compared to the default. > [!WARNING] > > `q4_0` KV cache introduces measurable quality loss on tasks that rely on precise attention over long context — needle-in-a-haystack retrieval, multi-document summarization, and long conversations. Test your specific workload before committing to `q4_0`. ## Thread configuration The `num_thread` parameter controls how many CPU threads Ollama uses for inference. The right value depends on how much work the CPU actually does. Set it in a Modelfile or per-request: ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3 PARAMETER num_thread 8 ``` ```bash API theme={"theme":{"light":"github-light","dark":"css-variables"}} curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3", "messages": [{"role": "user", "content": "Hello"}], "options": { "num_thread": 8 } }' ``` You can also set it globally with `OLLAMA_NUM_THREADS`: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Linux (systemd) sudo systemctl edit ollama.service # Add: Environment="OLLAMA_NUM_THREADS=8" sudo systemctl daemon-reload && systemctl restart ollama ``` | Scenario | Thread guidance | | - | - | | Full GPU offload (all layers on GPU) | `num_thread 2` to `num_thread 4` — the CPU handles orchestration and tokenization, not math | | Partial GPU offload (some layers on CPU) | Set `num_thread` to the number of physical cores on your CPU — not hyperthreads | | CPU-only (no GPU) | Set `num_thread` to the number of physical cores | > [!WARNING] > > More threads is not always faster. Past the number of physical cores, threads compete for cache and memory bandwidth and can slow down inference. When in doubt, start with the physical core count and benchmark from there. On Apple Silicon, thread tuning matters less because the GPU and CPU share unified memory and the MLX backend handles scheduling. The default usually works well. ## Batch size tuning The `num_batch` parameter controls how many tokens Ollama processes at once during the prefill phase (prompt ingestion). The default is 512. ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3 PARAMETER num_batch 1024 ``` | Setting | Effect | | - | - | | `num_batch 512` (default) | Balanced for interactive chat with short prompts | | `num_batch 1024` | Faster prefill for long prompts — roughly 50-100% faster time-to-first-token | | `num_batch 2048` | Maximum throughput for document ingestion — up to 150% faster prefill | | `num_batch 2048+` | Diminishing returns; higher memory usage without proportional speed gains | > [!TIP] > > For interactive chat with short prompts (under 1K tokens), the default `num_batch 512` is fine. Increase it only if you regularly ingest long documents and want faster time-to-first-token. > [!WARNING] > > Larger batch sizes consume more VRAM during prompt ingestion. If you increase `num_batch` and hit out-of-memory errors, reduce it or lower `num_ctx`. ## Parallel requests `OLLAMA_NUM_PARALLEL` controls how many requests a single model processes concurrently. The default is 1 — Ollama handles one request at a time per loaded model. ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Allow 4 parallel requests per model OLLAMA_NUM_PARALLEL=4 ``` | Value | Behavior | | - | - | | `1` (default) | One request at a time — lowest latency per request | | `2` to `4` | Multiple requests share the GPU — higher throughput, slightly higher per-request latency | | `8+` | Maximum throughput for batch workloads — requires significant VRAM for KV cache | > [!NOTE] > > RAM and VRAM scale with `OLLAMA_NUM_PARALLEL`. Each parallel request needs its own KV cache. If you serve 4 parallel requests with `num_ctx 8192`, you need 4× the KV cache memory compared to a single request. ### When to increase parallel requests Increase `OLLAMA_NUM_PARALLEL` when: * You serve multiple users or agents from the same model. * You run batch workloads where total throughput matters more than individual request latency. * You have VRAM headroom and the GPU is underutilized with single requests. Keep it at 1 when: * You serve a single user or agent. * You prioritize low latency over throughput. * VRAM is tight. ## Model memory management ### Keep models loaded with `OLLAMA_KEEP_ALIVE` By default, Ollama unloads a model from memory after 5 minutes of inactivity. The next request must reload the model from disk, which takes seconds to minutes depending on model size and disk speed. Set `OLLAMA_KEEP_ALIVE` to control how long models stay loaded: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Keep models loaded indefinitely OLLAMA_KEEP_ALIVE=-1 # Keep models loaded for 1 hour OLLAMA_KEEP_ALIVE=1h # Unload models immediately after each request OLLAMA_KEEP_ALIVE=0 ``` | Value | Behavior | | - | - | | `5m` (default) | Unload after 5 minutes of inactivity | | `-1` | Keep loaded indefinitely — no reload penalty | | `0` | Unload immediately after each request | | Duration (`1h`, `30m`, `24h`) | Keep loaded for that duration after the last request | > [!TIP] > > For a dedicated inference server that runs the same model all day, set `OLLAMA_KEEP_ALIVE=-1`. The model stays in memory and every request gets instant response with no cold-start penalty. ### Limit loaded models with `OLLAMA_MAX_LOADED_MODELS` Ollama can keep multiple models loaded simultaneously. The default is 3 × (number of GPUs) or 3 for CPU-only. If you serve many models and want to limit memory usage, lower this: ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Only keep 1 model loaded at a time OLLAMA_MAX_LOADED_MODELS=1 ``` When the limit is reached and a new model is requested, Ollama unloads the least-recently-used model to make room. ## Apple Silicon and MLX optimization Starting with Ollama 0.19, the default backend on Apple Silicon is Apple's MLX framework. MLX is optimized for Apple's unified memory architecture and delivers substantially faster inference than llama.cpp on M-series chips. ### Performance gains Benchmarks from Ollama's June 2026 MLX optimization update: | Model | Quantization | Tokens/sec (MLX) | | - | - | - | | Gemma 4 12B | NVFP4 | \~55 tok/s | | Gemma 4 12B | Q4\_K\_M | \~46 tok/s | | Llama 3.1 8B | Q4\_K\_M | \~70-80 tok/s | MLX also supports NVFP4 quantization on Apple Silicon, which delivers near-BF16 quality with 4-bit memory footprint — half the quality loss of Q4 quantization. ### Apple Silicon optimization tips 1. **Use MLX-optimized models.** When you pull a model on Apple Silicon, Ollama automatically uses the MLX backend. No configuration needed. 2. **Flash attention is always on.** MLX uses optimized attention kernels by default. The `OLLAMA_FLASH_ATTENTION` variable has no effect. 3. **Unified memory means full GPU offload.** Apple Silicon's GPU can access the full RAM pool, so `-ngl 999` (all layers on GPU) works as long as the model and KV cache fit in unified memory. 4. **Use `OLLAMA_KV_CACHE_TYPE=q8_0`** to stretch your memory budget. Halving the KV cache lets you run larger models or longer contexts. 5. **Set `OLLAMA_KEEP_ALIVE=-1`** for always-on inference. Models stay in unified memory with no reload penalty. > [!NOTE] > > If you need cross-platform portability or want to run the same configuration on NVIDIA GPU and Mac, you can force Ollama to use the llama.cpp backend on Apple Silicon. Contact Ollama support or check their documentation for the environment variable to override the backend selection. ## Example configurations ### Mac Studio M2 Ultra — 128 GB unified memory ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Environment variables OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=4 ``` ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3.1:70b-q4_K_M PARAMETER num_ctx 32768 PARAMETER num_batch 1024 ``` This configuration runs a 70B Q4 model with a 32K context window. The 128 GB unified memory pool handles the \~40 GB model weights, \~16 GB KV cache at q8\_0, and leaves headroom for 4 parallel requests. MLX delivers maximum throughput on Apple Silicon. ### NVIDIA RTX 4090 — 24 GB VRAM ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Environment variables OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=2 OLLAMA_NUM_THREADS=4 ``` ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3.1:8b-q4_K_M PARAMETER num_ctx 16384 PARAMETER num_batch 1024 ``` This configuration runs an 8B Q4 model with a 16K context window. The 24 GB VRAM handles the \~5 GB model weights, \~4 GB KV cache at q8\_0 with flash attention, and leaves headroom for 2 parallel requests. The 4 CPU threads handle orchestration and tokenization. ### Multi-GPU server — 4× RTX 4090 (96 GB total VRAM) ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Environment variables OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=8 OLLAMA_NUM_THREADS=8 ``` ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3.1:70b-q4_K_M PARAMETER num_ctx 8192 PARAMETER num_batch 2048 ``` This configuration runs a 70B Q4 model across 4 GPUs. Ollama automatically distributes layers across all available GPUs. The 96 GB total VRAM handles the \~40 GB model weights, \~16 GB KV cache, and leaves headroom for 8 parallel requests. The large batch size maximizes throughput for batch workloads. ### CPU-only server — 64 GB RAM, no GPU ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Environment variables OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=1 OLLAMA_NUM_THREADS=16 ``` ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3.1:8b-q4_K_M PARAMETER num_ctx 8192 PARAMETER num_batch 512 PARAMETER num_gpu 0 ``` This configuration runs an 8B Q4 model entirely on CPU. The 16 threads match the physical core count. The small batch size and single parallel request keep memory usage manageable. Expect 5-10 tok/s — usable for interactive chat but not batch workloads. ### High-throughput batch processing — RTX 4090 ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Environment variables OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q4_0 OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=8 OLLAMA_NUM_THREADS=4 ``` ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3.1:8b-q4_K_M PARAMETER num_ctx 4096 PARAMETER num_batch 2048 ``` This configuration maximizes throughput for offline batch workloads. The q4\_0 KV cache quarters memory usage, allowing 8 parallel requests in 24 GB VRAM. The large batch size accelerates prompt ingestion. The short 4K context keeps KV cache small. Expect 200-300 total tokens per second across all parallel requests. ### Low-latency interactive serving — RTX 4090 ```bash theme={"theme":{"light":"github-light","dark":"css-variables"}} # Environment variables OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_KEEP_ALIVE=-1 OLLAMA_NUM_PARALLEL=1 OLLAMA_NUM_THREADS=4 ``` ```dockerfile Modelfile theme={"theme":{"light":"github-light","dark":"css-variables"}} FROM llama3.1:8b-q4_K_M PARAMETER num_ctx 8192 PARAMETER num_batch 512 ``` This configuration minimizes latency for interactive chat. Single parallel request, q8\_0 KV cache for quality, and moderate batch size. Expect time-to-first-token under 500ms and 50-70 tok/s decode speed. ## Common mistakes and how to avoid them ### Mistake 1: Setting num\_ctx too high **Problem:** You set `num_ctx 128000` "just in case," but your prompts are under 2K tokens. You just wasted 64 GB of KV cache memory. **Fix:** Set `num_ctx` to match your actual workload. If your prompts are under 4K tokens, use `num_ctx 4096` or `num_ctx 8192`. You can always increase it later for specific requests via the API. ### Mistake 2: Forgetting flash attention **Problem:** You are using 30-50% more VRAM than necessary because flash attention is not enabled. **Fix:** Always set `OLLAMA_FLASH_ATTENTION=1`. There is no downside on supported hardware. ### Mistake 3: Not controlling model unload **Problem:** You restart your request after a 10-minute break, and the first request takes 30 seconds because Ollama reloaded the model from disk. **Fix:** Set `OLLAMA_KEEP_ALIVE=-1` for always-on inference. The model stays in memory and every request gets instant response. ### Mistake 4: Using too many threads on GPU **Problem:** You set `num_thread 32` on a 32-thread CPU, but all layers are on the GPU. The extra threads compete for cache and slow down orchestration. **Fix:** When all layers are on the GPU, use `num_thread 2` to `num_thread 4`. The CPU only handles orchestration and tokenization, not inference math. ### Mistake 5: Ignoring KV cache precision **Problem:** You are running out of VRAM and Ollama offloads layers to CPU, but you never considered that the KV cache is using 2× more memory than necessary. **Fix:** Set `OLLAMA_KV_CACHE_TYPE=q8_0` to halve KV cache memory. This is the best balance of quality and memory savings. ### Mistake 6: Setting OLLAMA\_NUM\_PARALLEL too high **Problem:** You set `OLLAMA_NUM_PARALLEL=16` on a 24 GB GPU, and now each request has only 1.5 GB of KV cache. Requests fail or quality degrades. **Fix:** Calculate your KV cache budget: `(VRAM - model_weights) / num_parallel`. If that leaves less than 1 GB per request, reduce `num_parallel` or lower `num_ctx`. ## Troubleshooting ### Ollama is slow after startup **Cause:** The model was unloaded and had to reload from disk. **Fix:** Set `OLLAMA_KEEP_ALIVE=-1` to keep the model loaded indefinitely. ### Out of memory errors **Cause:** The model + KV cache exceeds available VRAM or RAM. **Fix:** Reduce `num_ctx`, enable `OLLAMA_FLASH_ATTENTION=1`, set `OLLAMA_KV_CACHE_TYPE=q8_0`, or use a smaller quantization. ### GPU not being used **Cause:** Ollama did not detect the GPU, or VRAM is too small for even a few layers. **Fix:** Check `ollama ps` to see if layers are on GPU. For NVIDIA GPUs, ensure the NVIDIA drivers and CUDA toolkit are installed. For AMD GPUs, ensure ROCm is installed. ### Slow CPU inference **Cause:** Too many threads, or layers split between GPU and CPU. **Fix:** Set `num_thread` to the physical core count. If possible, offload more layers to GPU with `num_gpu`. Use a smaller quantization (Q4\_K\_M or Q5\_K\_M) to fit more layers on GPU. ## Next steps > **[Ollama remote access](/guides/ollama-remote-access.md)** > > Expose your Ollama model to teammates and agents through Tokios — without opening ports or setting OLLAMA\_HOST=0.0.0.0. > **[Hardware reference](/guides/hardware-reference.md)** > > Compare hardware platforms for local LLM inference — specs, model capacity, and recommendations. > **[Model fit finder](/guides/model-fit.md)** > > Find which models run on your hardware, or which hardware can run your target model. > **[Test your deployment](/guides/benchmark-your-deployment.md)** > > Measure latency and throughput after tuning to confirm the improvement. ## Frequently asked questions > **What is the best Ollama configuration for fast inference?** > > Enable flash attention (`OLLAMA_FLASH_ATTENTION=1`), set the context window to what your task actually needs (not the model maximum), use quantized KV cache (`OLLAMA_KV_CACHE_TYPE=q8_0` or `q4_0`) for long contexts, and make sure your model is fully GPU-offloaded. See the sections above for the exact commands. > **How do I reduce Ollama memory usage?** > > The 3 biggest levers: (1) lower `num_ctx` — memory scales linearly with context length, (2) quantize the KV cache with `OLLAMA_KV_CACHE_TYPE=q4_0` for up to 75% KV savings, (3) use a smaller quantization of the model weights (e.g., Q4\_0 instead of Q8\_0). Flash attention also saves 30-50% on attention memory. > **Does Ollama use MLX on Mac?** > > Yes — starting with Ollama 0.19, the default backend on Apple Silicon is Apple's MLX framework, not llama.cpp. MLX delivers substantially faster inference on M-series chips (55-80 tok/s on M2 Ultra vs 25-40 tok/s with llama.cpp). The `OLLAMA_FLASH_ATTENTION` variable has no effect on MLX because MLX always uses optimized attention kernels. > **How many parallel requests can Ollama handle?** > > Set `OLLAMA_NUM_PARALLEL` (default 1). Each parallel slot adds roughly 20-30% memory overhead for KV cache. For a 7B model at Q4 with 8K context, 4 parallel slots fit comfortably in 8 GB of VRAM. For larger models or longer contexts, start with 2 and measure. > **Why is Ollama slow to load my model?** > > Model load time is dominated by reading weights from disk and moving them to GPU. Use an NVMe SSD (not HDD), set `OLLAMA_KEEP_ALIVE` to keep the model warm between requests, and set `OLLAMA_MAX_LOADED_MODELS` to avoid evicting and reloading. On Apple Silicon, NVFP4 models load faster because the weights are smaller. > **What GPU offload settings should I use?** > > By default, Ollama auto-selects how many layers to offload to GPU. For maximum speed, set `num_gpu` to the model's total layer count (fully offload). If you run out of VRAM, reduce `num_ctx` before reducing `num_gpu` — keeping the model fully on GPU is almost always faster than partial offload. ## Sources * [Ollama FAQ](https://docs.ollama.com/faq) — official documentation on environment variables and configuration * [Ollama Modelfile reference](https://docs.ollama.com/modelfile) — parameter syntax and valid values * [Ollama MLX performance update](https://ollama.com/blog/mlx-performance) — June 2026 benchmarks and optimizations for Apple Silicon * [Ollama GPU scheduling deep dive](https://eastondev.com/blog/en/posts/ai/ollama-gpu-scheduling/) — VRAM management and multi-GPU behavior * [Ollama performance tuning: batching, KV cache, and OOM](https://eastondev.com/blog/en/posts/ai/20260410-ollama-performance-optimization/) — num\_batch benchmarks and memory calculations * [Ollama environment variables: complete reference](https://modelpiper.com/blog/ollama-environment-variables) — audited against v0.32.5 source > [!NOTE] > > After changing Ollama configuration, re-register your Tokios deployment so the connector picks up the new upstream endpoint. See [Register Model](/setup/register-model.md). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.