> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tokios.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Tokios exposes private local models through https://api.tokios.com using an outbound-only connector tunnel.
> Use 'connector' for the local tokios-connector process.
> Use 'deployment' for the public model name in the API model field.
> Use 'upstream model id' for the model served by Ollama, llama.cpp, vLLM, LM Studio, or another local backend.
> API keys use the prefix sk-tok-. Connector tokens use the prefix ct-tok-.

# Optimize llama.cpp for speed and memory

> Tune llama.cpp for maximum performance — flash attention, mmap vs mlock, thread tuning, GPU offload strategy, batch size, KV cache precision, and MoE expert placement.

llama.cpp is the most portable local inference runtime. It runs on CPU, NVIDIA GPU, AMD GPU, and Apple Silicon — often from the same binary. That flexibility comes with a long list of flags, and the defaults are not always optimal for your hardware. This guide covers the flags that matter most and the trade-offs behind each one.

> [!NOTE]
>
> Last verified: 2026-10-14 — llama.cpp build b11609.

> [!NOTE]
>
> All flags in this guide apply to `llama-server` and `llama-cli` from llama.cpp build **b11609** (October 14, 2026). Flag names can change between builds — run `llama-server --help` to confirm what your binary supports.

## Flash attention

Enable flash attention with `-fa` or `--flash-attn on`. In most benchmarks, this is the closest thing to a free performance improvement llama.cpp offers — you get faster inference with the same or lower memory usage, at a negligible quality cost.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -fa
```

Flash attention reworks how the model computes the attention scores between tokens. Instead of materializing the full attention matrix in memory, it computes attention in tiled chunks that fit in fast SRAM. The result:

* **Lower memory usage** for the KV cache — the biggest win at long context lengths.
* **Faster inference** — especially on GPU, where the tiled computation maps well to tensor cores.
* **Same output quality** — flash attention is mathematically equivalent to standard attention, not an approximation.

> [!TIP]
>
> **Always start with `-fa`.** There is no reason to leave it off unless you hit a compatibility issue with a specific model or platform. It is the single highest-impact flag in this guide.

## mmap vs mlock

llama.cpp loads model weights into memory using one of 2 strategies. The default is `--mmap`, which maps the model file directly into the process's virtual address space. The alternative is `--mlock`, which pins the entire model into physical RAM.

| Flag | What it does | Best for |
| - | - | - |
| `--mmap` (default) | Maps the file; the OS pages data in and out as needed | One-shot runs, memory-constrained machines, quick experiments |
| `--mlock` | Pins every byte in physical RAM; nothing pages out | Repeated inference, dedicated inference servers, models you run all day |

### When mmap wins

`--mmap` is the default because it works well for most interactive use. The OS loads model pages on demand, so startup is fast and the model file does not double-count against your RAM until the pages are actually touched. If you are testing models or running short sessions, the default is fine.

### When mlock wins

`--mlock` prevents the OS from evicting model pages to swap. That matters when:

* You run the same model all day and want consistent latency — no cold-start penalty from page faults.
* Your system is under memory pressure from other processes and might page out model weights.
* You are on a machine with slow swap (or no swap) and a page fault would stall inference for seconds.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -fa --mlock
```

> [!WARNING]
>
> `--mlock` requires that the full model fits in available physical RAM. If it does not fit, the process fails to start. Check your available memory with `free -h` on Linux or Activity Monitor on macOS before enabling it.

## Thread tuning

The `-t` flag sets the number of CPU threads used for inference. The right value depends on how much work the CPU actually does.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -t 8
```

| Scenario | Thread guidance |
| - | - |
| Full GPU offload (all layers on GPU) | `-t 2` to `-t 4` — the CPU handles orchestration and tokenization, not math |
| Partial GPU offload (some layers on CPU) | Set `-t` to the number of physical cores on your CPU — not hyperthreads |
| CPU-only (no GPU) | Set `-t` to the number of physical cores |

> [!TIP]
>
> More threads is not always faster. Past the number of physical cores, threads compete for cache and memory bandwidth and can slow down inference. When in doubt, start with the physical core count and benchmark from there.

On Apple Silicon, thread tuning matters less because the GPU and CPU share unified memory and the Metal backend handles scheduling. The default usually works well.

## GPU offload strategy

The `-ngl` flag controls how many model layers you offload to the GPU. More layers on the GPU means faster inference, but each layer consumes VRAM.

### Partial offload

Set `-ngl` to the number of layers you can fit in GPU memory. The remaining layers run on CPU.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -ngl 20 -fa
```

Use partial offload when the full model does not fit in VRAM. Start with a high value and reduce until the server starts without running out of memory.

### Full offload

Set `-ngl 999` to offload every layer. llama.cpp uses 999 as "all remaining layers" — it does not literally try to offload 999 layers.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -ngl 999 -fa
```

Full offload gives the best inference speed because every forward pass runs on the GPU. The model weights and KV cache must both fit in VRAM.

### Memory budgeting per layer

Each offloaded layer consumes VRAM for both weights and KV cache. A rough estimate:

```
VRAM per layer ≈ (parameter_count_per_layer × bytes_per_weight) + KV_cache_per_layer
```

For a 7B model at Q4\_K\_M (\~4.5 bits per weight, \~40 layers):

| Component | Per layer | 40 layers total |
| - | - | - |
| Weights | \~0.15 GB | \~6 GB |
| KV cache (f16, 4K ctx) | \~0.05 GB | \~2 GB |
| **Total** | | **\~8 GB** |

Leave 1-2 GB of VRAM headroom for CUDA context and overhead.

## Batch size tuning

Two flags control how many tokens llama.cpp processes at once during the prefill phase:

| Flag | What it controls |
| - | - |
| `-b` (batch size) | Total tokens processed in one batch during prompt ingestion |
| `-ub` (UBatch size) | Maximum tokens per micro-batch within a single batch |

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -b 512 -ub 512 -fa
```

### How they interact

* **`-b`** sets the outer limit. If a prompt has more tokens than `-b`, llama.cpp splits it into multiple batches.
* **`-ub`** sets the inner chunk size. llama.cpp processes the batch in micro-batches of up to `-ub` tokens.
* `-ub` must be less than or equal to `-b`.

### Trade-offs

| Setting | Effect |
| - | - |
| Larger `-b` (512-2048) | Faster prefill for long prompts — more tokens processed per forward pass |
| Larger `-b` | More VRAM/RAM consumed during prompt ingestion |
| Smaller `-b` (128-256) | Lower memory usage, slower prefill |
| `-ub` equal to `-b` | Simplest configuration; one micro-batch covers the full batch |
| `-ub` smaller than `-b` | Caps per-micro-batch memory; useful when `-b` is large but VRAM is tight |

> [!TIP]
>
> For interactive chat with short prompts (under 1K tokens), the default `-b 512` is fine. Increase it only if you regularly ingest long documents and want faster time-to-first-token.

## KV cache precision

The KV cache stores key and value tensors for every token in every active sequence. By default, llama.cpp uses f16 (16-bit floating point) for both keys and values. You can switch to quantized precision to cut memory usage at a small quality cost.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -fa \
  --cache-type-k q8_0 --cache-type-v q8_0
```

| Precision | Memory per element | Quality | When to use |
| - | - | - | - |
| `f16` (default) | 2 bytes | Baseline | Maximum quality; you have plenty of memory |
| `q8_0` | 1 byte | Near-identical to f16 | Best balance — halves KV cache memory with negligible quality loss |
| `q4_0` | 0.5 bytes | Slight degradation | Tight memory; long context or many concurrent sequences |

You can set keys and values independently. A common pattern is `q8_0` for both, which halves the KV cache footprint with less than 1% quality difference on most benchmarks.

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
# Aggressive: quarter the KV cache memory
llama-server -m model.gguf -fa \
  --cache-type-k q4_0 --cache-type-v q4_0
```

> [!WARNING]
>
> `q4_0` KV cache introduces measurable quality loss on tasks that rely on precise attention over long context — needle-in-a-haystack retrieval, multi-document summarization, and long conversations. Test your specific workload before committing to `q4_0`.

## MoE optimization

Mixture-of-experts models like Qwen3.5-35B-A3B or Mixtral store a large number of expert sub-networks in each layer, but only activate a few per token. The expert weights still live in memory even when inactive, which makes MoE models disproportionately large for their active parameter count.

On multi-GPU systems, you can place expert layers on the CPU when GPU memory is tight:

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf -ngl 999 --n-cpu-moe 4 -fa
```

`--n-cpu-moe <num>` keeps the specified number of expert sub-layers on the CPU while the rest of the model runs on GPU. This trades a small latency increase for fitting a larger model in VRAM.

| Scenario | Configuration |
| - | - |
| MoE fits entirely in VRAM | `-ngl 999` — no CPU experts needed |
| MoE almost fits, a few experts overflow | `--n-cpu-moe 2` to `--n-cpu-moe 8` |
| MoE is too large for any reasonable GPU setup | Combine `--n-cpu-moe` with partial `-ngl` |

> [!TIP]
>
> Start with `--n-cpu-moe 0` (all experts on GPU) and increase only when you run out of VRAM. Each expert moved to CPU adds latency because the data must cross the PCIe bus during the forward pass.

## Example configurations

### Apple Silicon — optimized for M-series Mac

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf \
  -fa \
  --mlock \
  -b 512 \
  -ub 512 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  -ngl 999 \
  --port 8080
```

This pins the model in unified memory, enables flash attention, and quantizes the KV cache to stretch your memory budget. Apple Silicon's unified memory means the GPU can access the full RAM pool, so full offload (`-ngl 999`) works as long as the model and KV cache fit.

> [!NOTE]
>
> llama.cpp runs well on Apple Silicon, but if you are deploying exclusively on Mac, [MLX](https://github.com/ml-explore/mlx) is optimized for Apple's Metal backend and may deliver faster inference for the same model. Use llama.cpp when you need cross-platform portability or an OpenAI-compatible server.

### GPU offload — NVIDIA or AMD discrete GPU

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf \
  -fa \
  -ngl 999 \
  -b 512 \
  -ub 512 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  -t 4 \
  --port 8080
```

Full GPU offload with flash attention and quantized KV cache. The 4 CPU threads handle orchestration and tokenization. Reduce `-ngl` if the model does not fit — start at the total layer count and drop by 5 until it loads.

### CPU-only — no GPU available

```bash theme={"theme":{"light":"github-light","dark":"css-variables"}}
llama-server -m model.gguf \
  -t 8 \
  -b 256 \
  -ub 256 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080
```

No GPU offload, so the CPU does all the work. Thread count matches physical cores. Smaller batch size keeps memory usage manageable. Use a Q4\_K\_M or Q5\_K\_M quantization for the model weights to keep the footprint small.

## Next steps

> **[llama.cpp remote access](/guides/llama-cpp-remote-access.md)**
>
> Expose your llama.cpp server to teammates and agents through Tokios — without opening ports.

> **[Speculative decoding](/guides/speculative-decoding.md)**
>
> Speed up decode with draft models, n-grams, and EAGLE — works with llama.cpp backends.

> **[Test your deployment](/guides/benchmark-your-deployment.md)**
>
> Measure latency and throughput after tuning to confirm the improvement.

> **[KV cache and context](/concepts/kv-cache-and-context.md)**
>
> Deeper explanation of how KV cache works across model architectures and runtimes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.