Optimize llama.cpp for speed and memory

llama.cpp is the most portable local inference runtime. It runs on CPU, NVIDIA GPU, AMD GPU, and Apple Silicon — often from the same binary. That flexibility comes with a long list of flags, and the defaults are not always optimal for your hardware. This guide covers the flags that matter most and the trade-offs behind each one.

Last verified: 2026-10-14 — llama.cpp build b11609.

All flags in this guide apply to llama-server and llama-cli from llama.cpp build b11609 (October 14, 2026). Flag names can change between builds — run llama-server --help to confirm what your binary supports.

Flash attention

Enable flash attention with -fa or --flash-attn on. In most benchmarks, this is the closest thing to a free performance improvement llama.cpp offers — you get faster inference with the same or lower memory usage, at a negligible quality cost.

llama-server -m model.gguf -fa

Flash attention reworks how the model computes the attention scores between tokens. Instead of materializing the full attention matrix in memory, it computes attention in tiled chunks that fit in fast SRAM. The result:

Lower memory usage for the KV cache — the biggest win at long context lengths.
Faster inference — especially on GPU, where the tiled computation maps well to tensor cores.
Same output quality — flash attention is mathematically equivalent to standard attention, not an approximation.

Always start with -fa. There is no reason to leave it off unless you hit a compatibility issue with a specific model or platform. It is the single highest-impact flag in this guide.

mmap vs mlock

llama.cpp loads model weights into memory using one of 2 strategies. The default is --mmap, which maps the model file directly into the process’s virtual address space. The alternative is --mlock, which pins the entire model into physical RAM.

Flag	What it does	Best for
`--mmap` (default)	Maps the file; the OS pages data in and out as needed	One-shot runs, memory-constrained machines, quick experiments
`--mlock`	Pins every byte in physical RAM; nothing pages out	Repeated inference, dedicated inference servers, models you run all day

When mmap wins

--mmap is the default because it works well for most interactive use. The OS loads model pages on demand, so startup is fast and the model file does not double-count against your RAM until the pages are actually touched. If you are testing models or running short sessions, the default is fine.

When mlock wins

--mlock prevents the OS from evicting model pages to swap. That matters when:

You run the same model all day and want consistent latency — no cold-start penalty from page faults.
Your system is under memory pressure from other processes and might page out model weights.
You are on a machine with slow swap (or no swap) and a page fault would stall inference for seconds.

llama-server -m model.gguf -fa --mlock

--mlock requires that the full model fits in available physical RAM. If it does not fit, the process fails to start. Check your available memory with free -h on Linux or Activity Monitor on macOS before enabling it.

Thread tuning

The -t flag sets the number of CPU threads used for inference. The right value depends on how much work the CPU actually does.

llama-server -m model.gguf -t 8

Scenario	Thread guidance
Full GPU offload (all layers on GPU)	`-t 2` to `-t 4` — the CPU handles orchestration and tokenization, not math
Partial GPU offload (some layers on CPU)	Set `-t` to the number of physical cores on your CPU — not hyperthreads
CPU-only (no GPU)	Set `-t` to the number of physical cores

More threads is not always faster. Past the number of physical cores, threads compete for cache and memory bandwidth and can slow down inference. When in doubt, start with the physical core count and benchmark from there.

On Apple Silicon, thread tuning matters less because the GPU and CPU share unified memory and the Metal backend handles scheduling. The default usually works well.

GPU offload strategy

The -ngl flag controls how many model layers you offload to the GPU. More layers on the GPU means faster inference, but each layer consumes VRAM.

Partial offload

Set -ngl to the number of layers you can fit in GPU memory. The remaining layers run on CPU.

llama-server -m model.gguf -ngl 20 -fa

Use partial offload when the full model does not fit in VRAM. Start with a high value and reduce until the server starts without running out of memory.

Full offload

Set -ngl 999 to offload every layer. llama.cpp uses 999 as “all remaining layers” — it does not literally try to offload 999 layers.

llama-server -m model.gguf -ngl 999 -fa

Full offload gives the best inference speed because every forward pass runs on the GPU. The model weights and KV cache must both fit in VRAM.

Memory budgeting per layer

Each offloaded layer consumes VRAM for both weights and KV cache. A rough estimate:

VRAM per layer ≈ (parameter_count_per_layer × bytes_per_weight) + KV_cache_per_layer

For a 7B model at Q4_K_M (~4.5 bits per weight, ~40 layers):

Component	Per layer	40 layers total
Weights	~0.15 GB	~6 GB
KV cache (f16, 4K ctx)	~0.05 GB	~2 GB
Total		~8 GB

Leave 1-2 GB of VRAM headroom for CUDA context and overhead.

Batch size tuning

Two flags control how many tokens llama.cpp processes at once during the prefill phase:

Flag	What it controls
`-b` (batch size)	Total tokens processed in one batch during prompt ingestion
`-ub` (UBatch size)	Maximum tokens per micro-batch within a single batch

llama-server -m model.gguf -b 512 -ub 512 -fa

How they interact

-b sets the outer limit. If a prompt has more tokens than -b, llama.cpp splits it into multiple batches.
-ub sets the inner chunk size. llama.cpp processes the batch in micro-batches of up to -ub tokens.
-ub must be less than or equal to -b.

Trade-offs

Setting	Effect
Larger `-b` (512-2048)	Faster prefill for long prompts — more tokens processed per forward pass
Larger `-b`	More VRAM/RAM consumed during prompt ingestion
Smaller `-b` (128-256)	Lower memory usage, slower prefill
`-ub` equal to `-b`	Simplest configuration; one micro-batch covers the full batch
`-ub` smaller than `-b`	Caps per-micro-batch memory; useful when `-b` is large but VRAM is tight

For interactive chat with short prompts (under 1K tokens), the default -b 512 is fine. Increase it only if you regularly ingest long documents and want faster time-to-first-token.

KV cache precision

The KV cache stores key and value tensors for every token in every active sequence. By default, llama.cpp uses f16 (16-bit floating point) for both keys and values. You can switch to quantized precision to cut memory usage at a small quality cost.

llama-server -m model.gguf -fa \
  --cache-type-k q8_0 --cache-type-v q8_0

Precision	Memory per element	Quality	When to use
`f16` (default)	2 bytes	Baseline	Maximum quality; you have plenty of memory
`q8_0`	1 byte	Near-identical to f16	Best balance — halves KV cache memory with negligible quality loss
`q4_0`	0.5 bytes	Slight degradation	Tight memory; long context or many concurrent sequences

You can set keys and values independently. A common pattern is q8_0 for both, which halves the KV cache footprint with less than 1% quality difference on most benchmarks.

# Aggressive: quarter the KV cache memory
llama-server -m model.gguf -fa \
  --cache-type-k q4_0 --cache-type-v q4_0

q4_0 KV cache introduces measurable quality loss on tasks that rely on precise attention over long context — needle-in-a-haystack retrieval, multi-document summarization, and long conversations. Test your specific workload before committing to q4_0.

MoE optimization

Mixture-of-experts models like Qwen3.5-35B-A3B or Mixtral store a large number of expert sub-networks in each layer, but only activate a few per token. The expert weights still live in memory even when inactive, which makes MoE models disproportionately large for their active parameter count. On multi-GPU systems, you can place expert layers on the CPU when GPU memory is tight:

llama-server -m model.gguf -ngl 999 --n-cpu-moe 4 -fa

--n-cpu-moe <num> keeps the specified number of expert sub-layers on the CPU while the rest of the model runs on GPU. This trades a small latency increase for fitting a larger model in VRAM.

Scenario	Configuration
MoE fits entirely in VRAM	`-ngl 999` — no CPU experts needed
MoE almost fits, a few experts overflow	`--n-cpu-moe 2` to `--n-cpu-moe 8`
MoE is too large for any reasonable GPU setup	Combine `--n-cpu-moe` with partial `-ngl`

Start with --n-cpu-moe 0 (all experts on GPU) and increase only when you run out of VRAM. Each expert moved to CPU adds latency because the data must cross the PCIe bus during the forward pass.

Example configurations

Apple Silicon — optimized for M-series Mac

llama-server -m model.gguf \
  -fa \
  --mlock \
  -b 512 \
  -ub 512 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  -ngl 999 \
  --port 8080

This pins the model in unified memory, enables flash attention, and quantizes the KV cache to stretch your memory budget. Apple Silicon’s unified memory means the GPU can access the full RAM pool, so full offload (-ngl 999) works as long as the model and KV cache fit.

llama.cpp runs well on Apple Silicon, but if you are deploying exclusively on Mac, MLX is optimized for Apple’s Metal backend and may deliver faster inference for the same model. Use llama.cpp when you need cross-platform portability or an OpenAI-compatible server.

GPU offload — NVIDIA or AMD discrete GPU

llama-server -m model.gguf \
  -fa \
  -ngl 999 \
  -b 512 \
  -ub 512 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  -t 4 \
  --port 8080

Full GPU offload with flash attention and quantized KV cache. The 4 CPU threads handle orchestration and tokenization. Reduce -ngl if the model does not fit — start at the total layer count and drop by 5 until it loads.

CPU-only — no GPU available

llama-server -m model.gguf \
  -t 8 \
  -b 256 \
  -ub 256 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080

No GPU offload, so the CPU does all the work. Thread count matches physical cores. Smaller batch size keeps memory usage manageable. Use a Q4_K_M or Q5_K_M quantization for the model weights to keep the footprint small.

Next steps

llama.cpp remote access

Expose your llama.cpp server to teammates and agents through Tokios — without opening ports.

Speculative decoding

Speed up decode with draft models, n-grams, and EAGLE — works with llama.cpp backends.

Test your deployment

Measure latency and throughput after tuning to confirm the improvement.

KV cache and context

Deeper explanation of how KV cache works across model architectures and runtimes.

Hardware & fit

Serve & connect your runtime

Connect a client or app

Compare

Optimize llama.cpp for speed and memory

Flash attention

mmap vs mlock

When mmap wins

When mlock wins

Thread tuning

GPU offload strategy

Partial offload

Full offload

Memory budgeting per layer

Batch size tuning

How they interact

Trade-offs

KV cache precision

MoE optimization

Example configurations

Apple Silicon — optimized for M-series Mac

GPU offload — NVIDIA or AMD discrete GPU

CPU-only — no GPU available

Next steps

llama.cpp remote access

Speculative decoding

Test your deployment

KV cache and context

​Flash attention

​mmap vs mlock

​When mmap wins

​When mlock wins

​Thread tuning

​GPU offload strategy

​Partial offload

​Full offload

​Memory budgeting per layer

​Batch size tuning

​How they interact

​Trade-offs

​KV cache precision

​MoE optimization

​Example configurations

​Apple Silicon — optimized for M-series Mac

​GPU offload — NVIDIA or AMD discrete GPU

​CPU-only — no GPU available

​Next steps

llama.cpp remote access

Speculative decoding

Test your deployment

KV cache and context

Flash attention

mmap vs mlock

When mmap wins

When mlock wins

Thread tuning

GPU offload strategy

Partial offload

Full offload

Memory budgeting per layer

Batch size tuning

How they interact

Trade-offs

KV cache precision

MoE optimization

Example configurations

Apple Silicon — optimized for M-series Mac

GPU offload — NVIDIA or AMD discrete GPU

CPU-only — no GPU available

Next steps